Module 5
Checking extraction
By the end of this module you can check an extracted obligations table in both directions, name the five kinds of extraction error, work out row precision and coverage, and plan a spot-check of a large batch.
Part 1. The lecture
What extraction does
Extraction tools read contracts and fill a table: who must do what, by when, for how much. That table then feeds a calendar, a risk register or a due-diligence report. Everything downstream trusts it, which is why a quiet error in the table is more dangerous than a visible one in a summary.
Five kinds of error
- Wrong value: a figure, date or period is changed (1.5% per month becomes per year).
- Wrong unit or basis: the number is right but the measure is not (business days become calendar days).
- Wrong party: the duty is given to the other side.
- Invented row: an obligation with no basis in the source, often with a plausible clause reference.
- Omission: a real obligation never reaches the table.
The first three look right at a glance. The last is invisible unless you go looking for it.
Check in both directions
- Table to source. For each row, find the clause it cites and read it. Check the party, the obligation, the figure and the unit. Open the clause: do not trust the reference.
- Source to table. Read the source clause by clause and ask of each, “Is there a row for this?” This is the only way to find omissions.
Two numbers for a table
Row precision is the share of rows that are fully correct: correct rows divided by all rows. Coverage is the recall of the extraction: obligations correctly captured divided by obligations in the source. Take a table with 40 rows, 30 correct, from a source with 50 obligations: precision is 75% and coverage is 60%. Report both. A tool can be precise and still miss half the contract.
Spot-checking a large batch
You cannot check 500 tables by hand. Draw a random sample of contracts (Module 3) and check every field in them against the source. Then estimate the error rate for each field type with a margin. Amounts, dates, parties and units deserve the closest look, because a slip there changes money or rights. Check scans, tables and annexes first, because that is where extraction most often fails.
Part 2. Lab: Extraction Checker
A tool read the carriage agreement below and produced an obligations table. Compare the table with the source. Mark each row that is wrong, and tick each source clause whose obligation the table leaves out. There are six problems to find.
Source agreement (Alderwick Exports and Tessaly Freight)
- 1.Services. Tessaly shall carry Alderwick's goods between the ports named in Schedule 1.
- 2.Payment. Alderwick shall pay each invoice within 30 days of receiving it.
- 3.Late payment. Any sum not paid when due bears interest at 1.5% per month until it is paid.
- 4.Insurance. Tessaly shall maintain cargo insurance of at least $2,000,000 throughout the term.
- 5.Delivery. Tessaly shall deliver the goods within 10 business days of collecting them.
- 6.Delay. Tessaly shall tell Alderwick within 24 hours of becoming aware of any delay longer than 2 days.
- 7.Liability. Tessaly's total liability for any one claim is limited to $500,000.
- 8.Confidentiality. Each party shall keep the other's pricing confidential for 3 years after this agreement ends.
- 9.Termination. Either party may end this agreement by giving 60 days' written notice.
- 10.Audit. Alderwick may audit Tessaly's delivery records once a year on 14 days' notice.
The tool’s obligations table
| # | Party | Obligation | Detail | Clause | Wrong? |
|---|---|---|---|---|---|
| 1 | Alderwick | Pay invoices | Within 30 days of receipt | 2 | |
| 2 | Alderwick | Pay interest on overdue sums | 1.5% per year | 3 | |
| 3 | Tessaly | Maintain cargo insurance | At least $2,000,000 throughout the term | 4 | |
| 4 | Tessaly | Deliver the goods | Within 10 calendar days of collection | 5 | |
| 5 | Alderwick | Notify of any delay longer than 2 days | Within 24 hours | 6 | |
| 6 | Tessaly | Liability limited per claim | $500,000 | 7 | |
| 7 | Both parties | Keep the other's pricing confidential | 3 years after the agreement ends | 8 | |
| 8 | Either party | May end on written notice | 60 days | 9 | |
| 9 | Tessaly | Keep ISO 27001 certification | Throughout the term | 11 |
The agreement, the parties and the table are invented for practice.
Part 3. Check your understanding
Ten questions drawn from a larger pool. In this pilot your score is shown but not saved.
This course teaches how to test and supervise tools, using invented documents, fictional parties and made-up tool outputs. No real tool is run, and nothing here is legal advice. Never paste a real client’s confidential material into a public tool.