Module 2
Clause libraries and tagging
By the end of this module you can design a small taxonomy for tagging clauses, write definitions that two people will apply the same way, and measure how well they do with percent agreement and Cohen's kappa.
Part 1. The lecture
What a clause library is for
A clause library files each clause under a tag. Once clauses are tagged you can find every termination clause in the portfolio, compare how liability is limited, count how many contracts have a confidentiality duty, and draft from the best examples. It also underpins any tool that claims to find clauses automatically, because the tags are its answer key.
Designing the taxonomy
- Start small. Eight to fifteen tags cover most business contracts. Every extra tag gives reviewers another chance to disagree.
- Make the tags mutually distinct. If reviewers cannot tell two tags apart, merge them.
- Decide how to treat mixed clauses. A clause that both caps liability and promises to cover third-party claims needs a rule: allow two tags, or name which one wins.
- Tag at the level you will count. If you will only ever ask “is there a termination clause?”, you do not need ten kinds of termination clause.
Definitions do the work
Each tag needs a one-sentence rule and a couple of examples, including borderline ones. For instance: Termination: a right to end the whole agreement before the end of its term, with or without cause. Not: the length of the term, or renewal. A tag with no written rule means different things to different reviewers, and your counts then measure the reviewers instead of the contracts.
Measuring agreement
To test the tags, have two reviewers tag the same clauses independently, then compare. Percent agreement is the share of clauses on which they chose the same tag. If they agree on 16 of 20, that is 80%.
Percent agreement flatters, because two people choosing among only a few tags will sometimes match by luck. Cohen’s kappa discounts that chance matching. It is 1 for perfect agreement and around 0 for agreement no better than chance. As a rough guide, a kappa below about 0.6 means the tags are too loose to count on.
Fixing disagreement
- List the disagreements and look for the pattern. They almost always cluster on one or two tag pairs.
- Find the missing sentence in the definitions, and add it, with an example of the borderline case.
- Tag a fresh batch and measure again. Improvement on the clauses you studied proves nothing.
- Remember what agreement does not show. Two reviewers can share the same wrong reading. Check some tags against a trusted answer key too.
Part 2. Lab: Clause Tagger
Part A. Tag the clauses
A clause library files each clause under one tag. Give each clause below the tag you think fits best from the eight in the taxonomy.
- 1.Each invoice is payable within 30 days of receipt.
- 2.Overdue sums bear interest at 1.5% per month.
- 3.The agreement renews for successive 12-month periods unless a party gives 90 days' notice.
- 4.Either party may end this agreement on 60 days' written notice.
- 5.Supplier's total liability is limited to $500,000 for any one claim.
- 6.Neither party is liable for indirect or consequential loss.
- 7.Supplier shall compensate Buyer for any loss arising from a claim that the goods infringe a third party's patent.
- 8.Each party shall keep the other's pricing confidential for 3 years.
- 9.Neither party may transfer its rights under this agreement without written consent.
- 10.This agreement is governed by the laws of Singapore.
- 11.Buyer may terminate immediately if Supplier becomes insolvent.
- 12.The initial term is 2 years from the effective date.
Part B. How consistent are two reviewers?
Two reviewers tagged the same 20 clauses independently. Their tags are below. Where they differ, a count of “Termination clauses” depends on who tagged the contract.
| # | Reviewer A | Reviewer B | |
|---|---|---|---|
| 1 | Payment | Payment | |
| 2 | Payment | Payment | |
| 3 | Term and renewal | Term and renewal | |
| 4 | Termination | Term and renewal | differ |
| 5 | Liability limits | Liability limits | |
| 6 | Liability limits | Indemnity | differ |
| 7 | Indemnity | Indemnity | |
| 8 | Confidentiality | Confidentiality | |
| 9 | Assignment | Assignment | |
| 10 | Governing law | Governing law | |
| 11 | Termination | Termination | |
| 12 | Term and renewal | Term and renewal | |
| 13 | Termination | Term and renewal | differ |
| 14 | Payment | Payment | |
| 15 | Indemnity | Liability limits | differ |
| 16 | Confidentiality | Confidentiality | |
| 17 | Governing law | Governing law | |
| 18 | Liability limits | Liability limits | |
| 19 | Assignment | Assignment | |
| 20 | Term and renewal | Term and renewal |
Percent agreement
80%
They chose the same tag for 16 of 20 clauses.
Cohen’s kappa
0.77
Agreement beyond chance: it allows for two reviewers sometimes matching by luck.
- Clause 4: A said Termination, B said Term and renewal. 'Either party may end on notice' was tagged Termination by one reviewer and Term and renewal by the other. The definition of Termination did not say whether it covers ending a contract without cause.
- Clause 6: A said Liability limits, B said Indemnity. An exclusion of indirect loss was tagged Liability limits by one reviewer and Indemnity by the other. The definitions did not say how to tell a limit on liability from a promise to cover loss.
- Clause 13: A said Termination, B said Term and renewal. Another 'ending on notice' clause, split the same way as item 4. A single sentence in the definition would settle both.
- Clause 15: A said Indemnity, B said Liability limits. A promise to pay a capped sum for third-party claims is both an indemnity and a limit. The definition has to say which tag wins when a clause does both.
Every disagreement comes from a missing sentence in a definition, not from carelessness. The fix is to tighten the definitions, tag a fresh batch, and measure agreement again. A rule of thumb is that a kappa under about 0.6 means the tags are too loose to count on.
Part 3. Check your understanding
Ten questions drawn from a larger pool. In this pilot your score is shown but not saved.
This course teaches how to turn contracts into reliable data, using invented contracts, fictional parties and made-up figures. No real contracts or tools are used, and nothing here is legal advice. Never paste a real client’s confidential contracts into a public tool.