Module 2
Measuring a tool
By the end of this module you can fill in a confusion table, work out accuracy, precision, recall and elusion, say which one matters for a given job, and explain why a tool that flags nothing can score 99 per cent on accuracy.
Part 1. The lecture
The confusion table
To measure a review tool you need an answer key: a person has already decided which items really are problems. Then every item falls into one of four boxes, depending on two facts: was it really a problem? and did the tool flag it?
| Tool flagged | Tool did not flag | |
|---|---|---|
| Really a problem | True positive (a catch) | False negative (a miss) |
| Not a problem | False positive (a false alarm) | True negative (rightly left alone) |
Four measures
- Accuracy: right calls out of all items. Easy to read, easy to be fooled by.
- Precision: of everything flagged, the share that was a real problem. It tells you how much reviewers’ time the flags will waste.
- Recall: of all the real problems, the share the tool found. It tells you what slipped through. (Also called sensitivity.)
- Elusion: of the items the tool left alone, the share that were in fact real problems. It tells you how clean the “fine” pile is.
Worked example: 8 true positives, 2 false positives, 4 false negatives, 86 true negatives (100 items). Accuracy is 94 / 100 = 94%. Precision is 8 / 10 = 80%. Recall is 8 / 12 = 67%. Elusion is 4 / 90 = 4.4%.
Why accuracy misleads
Suppose 2 per cent of documents hold a real problem. A tool that flags nothing is right about the 98 per cent that are fine, so its accuracy is 98 per cent. It is also worthless. Accuracy is dominated by the large pile of easy cases, so when real problems are rare it says almost nothing about the tool’s real job.
Rarity hurts precision too. If a tool wrongly flags 5 per cent of clean documents and there are 980 of them, that is about 49 false alarms, against perhaps 18 real catches. Most flags are then wrong, even for a good tool.
Which measure matters?
- A missed problem is costly (a hidden indemnity, a missed deadline): watch recall and elusion.
- Reviewers’ time is the constraint and a flag costs an hour: watch precision.
- Both matter, and improving one often hurts the other. A stricter tool catches more and raises more false alarms. Choose where to sit on purpose, and write it down.
Reading a vendor’s numbers
- If the figure is accuracy, ask for recall and precision.
- Ask what documents it was tested on, how they were chosen, and whether they resemble yours.
- Ask who decided what the right answer was, and how many reviewers agreed.
- Remember that the figures on a vendor’s own page are marketing until you have tested the tool on your own sample (Module 3).
Common mistakes
- Quoting accuracy when problems are rare.
- Mixing up precision and recall.
- Counting a tool’s own output as the answer key.
- Measuring on an easy set and expecting the same result on a hard one.
- Reporting one measure without the others.
Part 2. Lab: Review Scorer
A reviewer marked each of these 24 clauses as risky or fine. A review tool flagged some of them. Count the four groups, then check your counts.
| # | Clause | Reviewer says | Tool says |
|---|---|---|---|
| 1 | Unlimited liability for the Client | Risky | Flagged |
| 2 | Standard 30-day payment term | Fine | Not flagged |
| 3 | Automatic renewal with a 6-month notice window | Risky | Flagged |
| 4 | Governing law of the Supplier's home country | Fine | Flagged |
| 5 | Entire agreement clause | Fine | Not flagged |
| 6 | Supplier may change prices at any time without notice | Risky | Flagged |
| 7 | Confidentiality for five years after the term | Fine | Not flagged |
| 8 | Assignment needs the other party's consent | Fine | Flagged |
| 9 | Client indemnifies Supplier for all losses, with no cap | Risky | Not flagged |
| 10 | Notices by email are valid | Fine | Not flagged |
| 11 | Force majeure covers any event the Supplier chooses | Risky | Flagged |
| 12 | Counterparts and electronic signature | Fine | Not flagged |
| 13 | Headings do not affect interpretation | Fine | Not flagged |
| 14 | Supplier may subcontract without limit | Risky | Not flagged |
| 15 | Invoices to be sent by the 5th of each month | Fine | Flagged |
| 16 | Severability clause | Fine | Not flagged |
| 17 | Client must give 90 days' notice to end early | Fine | Not flagged |
| 18 | Waiver must be in writing | Fine | Not flagged |
| 19 | Data processed only on the Client's instructions | Fine | Not flagged |
| 20 | Term of twelve months from the start date | Fine | Not flagged |
| 21 | Definitions section | Fine | Not flagged |
| 22 | Supplier insurance of at least USD 1,000,000 | Fine | Flagged |
| 23 | Dispute escalation to senior managers before court | Fine | Not flagged |
| 24 | Third party rights excluded | Fine | Not flagged |
The reviewer column is the human answer key. Every figure here is invented for practice.
Part 3. Lab: Why Accuracy Misleads
Imagine 1,000 documents. Choose how common real problems are, and how well the tool performs. Then compare it with a useless tool that never flags anything.
| Tool | Accuracy | Precision | Recall | Real problems missed |
|---|---|---|---|---|
| The tool | 94.9% | 27% | 90% | 2 of 20 |
| A tool that flags nothing at all | 98.0% | n/a | 0% | 20 of 20 |
With real problems in 2% of documents, the tool that does nothing scores 98.0% on accuracy while missing every real problem. Your tool scores 94.9% on accuracy, which is lower than the tool that does nothing, yet it finds 90% of the real problems. Accuracy and usefulness have come apart.
Notice also the precision of 27%: about 73% of its flags are false alarms, because real problems are rare and a small false-positive rate on many clean documents adds up. After the tool has looked, about 0.2% of the unflagged pile still holds a real problem.
So never judge a review tool by accuracy alone. Ask for recall, precision and elusion, measured on a sample you drew yourself.
Part 4. Check your understanding
Ten questions drawn from a larger pool. In this pilot your score is shown but not saved.
This course teaches how to test and supervise tools, using invented documents, fictional parties and made-up tool outputs. No real tool is run, and nothing here is legal advice. Never paste a real client’s confidential material into a public tool.