Module 4
Acceptance-rate analytics
By the end of this module you can read an acceptance dashboard, break it down by clause type, spot the signs of reviewers clicking Accept without reading, and use the data to decide where to sample and test.
Part 1. The lecture
What the log records
Many drafting and review tools log what happens to each suggestion: it was accepted as written, edited and then used, or rejected. Vendors publish the accepted share as evidence that their tool works. In-house teams can pull the same log for their own use. It is a rich, free source of data, and an easy one to misread.
It measures behaviour, not correctness
An acceptance rate counts what reviewers did. It does not say whether the suggestions were right. Two readings are always possible:
- High acceptance: the tool is excellent at this clause, or reviewers have stopped checking it.
- Low acceptance: the tool is weak here, or reviewers have a strong style preference that the tool does not share.
The rate cannot tell the two apart. That takes a look at the suggestions themselves.
Break it down
An overall figure is an average of very different situations. A tool may be superb at confidentiality clauses and poor at indemnities, and the average hides it. Split the log by clause type, document type, reviewer and period. A change in the overall rate may come from a change in the mix (more easy clauses this quarter) and not from any change in the tool.
Add time and reasons
- Time spent. A 97% acceptance rate at six seconds a suggestion is a different thing from 97% at ninety seconds. The first deserves a sample before anyone celebrates.
- Edited-then-used. Keep this as its own category. The tool helped, but needed correcting. Counting it as a full acceptance flatters the tool, and counting it as a rejection ignores the saving.
- Reason codes. Ask reviewers to say why they rejected: wrong party, wrong figure, wrong clause, or style only. Counts say how often, and reasons say why.
What to do with it
Use acceptance data as a map, not a verdict. It covers every suggestion, so it shows cheaply where the trouble may be. Then take a random sample of suggestions in the interesting places, both the weak ones and the suspiciously smooth ones, and check them against the source as in Module 3. When a vendor quotes an acceptance figure, ask: accepted by whom, on which clauses, in how many seconds, and were the accepted suggestions ever checked?
Part 2. Lab: Acceptance Dashboard
Alderwick’s legal team used a drafting tool for 180 suggested changes last quarter. Each suggestion was accepted as it stood, edited and then used, or rejected. The dashboard headline says 66% accepted and 87% usable. Is that good news? Look underneath.
| Mix | |||||
|---|---|---|---|---|---|
| Assignment | 15 | 73% | 7% | 27 | |
| Confidentiality | 30 | 97% | 0% | 6 | |
| Force majeure | 15 | 67% | 13% | 40 | |
| Governing law | 20 | 90% | 5% | 20 | |
| Indemnity | 25 | 28% | 32% | 109 | |
| Limitation of liability | 20 | 40% | 25% | 91 | |
| Payment terms | 30 | 80% | 7% | 43 | |
| Termination | 25 | 48% | 20% | 67 |
Mix bar: green accepted as is, gold edited then used, red rejected. Click a heading to sort. The suggestion log behind these figures is invented for practice.
Part 3. Check your understanding
Ten questions drawn from a larger pool. In this pilot your score is shown but not saved.
This course teaches how to test and supervise tools, using invented documents, fictional parties and made-up tool outputs. No real tool is run, and nothing here is legal advice. Never paste a real client’s confidential material into a public tool.