The Law To Know

Module 7

Reading results honestly

By the end of this module you can name the six commonest ways a contract dashboard misleads without any arithmetic being wrong, ask the five questions that expose them, and report a result with its limits.

Part 1. The lecture

Six patterns

  1. An extreme value drags the average. One 12-times cap lifts the average cap above what any typical contract has. Report the median, and name the outlier.
  2. Missing data counted as “no”. If 7 of 60 caps are not recorded, a share of “uncapped” that treats them as capped is understated. Show unknowns separately or give a range.
  3. Mixed units. Euros added to dollars, weeks to days. Convert at a stated rate and date, or total each unit separately.
  4. An unrepresentative sample. The contracts one office has uploaded are not a random slice of the portfolio. Say what the data covers and do not generalise beyond it.
  5. A percentage with no count. “Disputes fell 100%” may mean one dispute became none. Always show the base.
  6. A scale that exaggerates. A bar chart that starts at 90% turns a gap of two points into a cliff. Bars should start at zero.

Five questions to ask of any figure

  1. What is it a count or total of?
  2. What was left out, and how many?
  3. What was assumed (what does a blank mean, which exchange rate)?
  4. Would the answer change if I used the whole portfolio?
  5. Is the base large enough to tell me anything?

Pattern is not cause

Contracts with arbitration clauses may show fewer court disputes. That does not show the clause prevents disputes. Contracts with an arbitration clause may also be larger, with more careful counterparties. Analytics can point to where to look. A claim about cause needs more than a pattern.

When data is incomplete, give a range

If the question asks for one number and the data cannot support it, say so and give a range: “between 15% and 27% of contracts are uncapped, depending on 7 contracts we have not yet checked.” A range is honest about what is unknown and shows what would settle it. It also protects you if the missing records turn out to be unlike the rest.

If a tool extracted the data

Any error rate in extraction carries into every result. Apply the sampling method from Checking Legal AI Tools to your data set: take a random sample of records, check them against the source contracts, estimate the error rate with its margin, and report it with the results.

Part 2. Lab: Dashboard Critic

Four dashboards from Alderwick’s legal operations team each make a confident claim. Each has one flaw in how the data was used. Name the flaw. The same six flaws are offered for every dashboard.

Dashboard 1: Typical liability cap

The average liability cap across the portfolio is 1.6× annual fees. We are generally well protected.

Based on 44 contracts with a numeric cap. The largest single cap is 12× fees. The median cap is 1×.

Dashboard 2: Uncapped liability

Only 15% of contracts (9 of 60) have unlimited liability. Low risk.

7 contracts have no cap recorded. They have been counted as capped.

Dashboard 3: Spend up for renewal

Annual value of contracts reaching the end of their term in the next 12 months: $2,096,000.

Three of these contracts are priced in euros. Their euro amounts have been added to the dollar totals without conversion.

Dashboard 4: Governing law

English law governs 50% of our contracts. We can standardise on English-law templates.

Based on the 24 contracts uploaded so far by the London office. The other 36 are still in paper files elsewhere.

Part 3. Check your understanding

Ten questions drawn from a larger pool. In this pilot your score is shown but not saved.

Loading quiz…

This course teaches how to turn contracts into reliable data, using invented contracts, fictional parties and made-up figures. No real contracts or tools are used, and nothing here is legal advice. Never paste a real client’s confidential contracts into a public tool.