Sampling and validation

Module 2 gave you the measures. To compute them you need an answer key, and an answer key means a person reading the document. Doing that for every document defeats the point of a tool. A random sample solves it: check a modest number of documents by hand, compute the measures on those, and use them to estimate the measures for the rest.

The estimate comes with a margin. The margin is not a failing of the method. It is the honest statement of how much the sample can tell you.

In a random sample every document has the same chance of being chosen, and chance alone decides. Pick the first hundred in the folder, or the ones that look interesting, or the ones a colleague suggests, and you have a convenience sample. Such samples are usually unrepresentative in ways you cannot see: the first documents are the oldest, and the interesting ones are the unusual ones.

Number the documents, then let a random-number generator, a spreadsheet or a dice-and-table method choose the numbers. Write down how you did it.

Suppose 100 flagged contracts are checked and 80 prove to be real problems, so the sample precision is 80%. If you drew a new sample, you would not get exactly 80% again. A rough 95% margin is 1.96 × √(p × (1 − p) / n). Here that is 1.96 × √(0.8 × 0.2 / 100), about 0.078. So the true precision is probably between about 72% and 88%.

A caution: when a result is at or near 0% or 100%, or the sample is small, this simple formula wrongly shrinks toward nothing. A tool that finds 13 of 13 real problems has not been proved perfect. A common fix is to add two successes and two failures before working out the margin, and the lab below does that.

Two facts to remember. First, the margin falls with the square root of the sample, so to halve it you need four times as many documents. Second, the n that matters is the number of cases the measure is about: flagged documents for precision, and real problems for recall.

Recall asks: of the real problems, how many did the tool find? If real problems are in 5% of documents, a sample of 100 holds about five of them. Recall measured on five cases can swing wildly: finding four of five gives 80%, and finding three of five gives 60%. The sample was 100 documents, but the recall estimate rests on five.

There are two practical cures. Draw a much larger sample, or check the unflagged pile separately: take a sample of the documents the tool left alone and count the real problems in it. That count, scaled up to the size of the pile, tells you how many the tool missed.

• A vendor’s test is about its documents. Yours may be longer, older, in a different style or from a different jurisdiction. Only your own sample describes your work.
• Set the answer key blind. The person deciding what counts as a real problem should not have seen the tool’s flags first, or they will tend to agree with it.
• Re-test after changes. An update, a new document type or a new instruction can change results. A result belongs to a version, a task and a time.

Compare the whole range, not the point estimate, with your target. If the target is 85% and your recall is 80% give or take 12, the range runs from 68% to 92% and contains the target. You cannot say the tool passes or fails. The honest answers are collect more evidence or decide on a different ground, such as the cost of a miss. A confident go or no-go from a range that straddles the line is a guess presented as a finding.