How Well Does Your Cite Checker Stack Up? LawDiver's 5,300-Citation Benchmark
· LawDiver
5,300 citations across six accuracy bars — including separate scores for easier Bluebook mangling and harder recovery problems. Here is how CiteCheck did.
5,300 citations. One public suite. Six accuracy bars.
If you sell a cite checker — or you are about to buy one — you need more than a marketing claim that “it finds hallucinations.” Real litigation work mixes clean authorities, Bluebook noise, OCR wreckage, neighbor-page identity traps, and cases that are still good on the books but no longer good law. A checker that only declines fake cites can still hurt you by silently confirming an overruled decision, or by calling a typo’d real case “not found” without recovering what the author meant.
LawDiver built a single combined benchmark of 5,300 citation strings for that reality. The inputs and answer key are public so anyone can run the same set against their own product. This article explains how the suite is graded and how CiteCheck performed on it.
Public test + answer key: benchmarks/citecheck.
What the suite is for
Cite checking is not one skill. Confirming a perfect Supreme Court cite is easy. Declining a volume that cannot exist is also easy for a competent corpus. The hard parts are the ones lawyers actually trip over:
- A real case that has been overruled, where confirming the caption without mentioning treatment is worse than a miss.
- A fabricated cite that looks almost right — wrong parties on a real reporter page, invented series, impossible WL numbers.
- A clean cite that must not be rejected just because the product’s coverage is incomplete or picky about form.
- A cite that is still the same case but written with Bluebook or form noise — spacing, abbreviations, court parentheticals.
- A cite that is a real case badly damaged — transposed volume digits, wrong first page, OCR letter swaps, structural breakage — where the right answer is to recover the intended authority and flag the defect.
The 5,300-string suite mixes all of those into one shuffled list covering all fifty states, federal courts, specialty dockets, and non-case authorities. Overruled rows sit in the same shuffle as fabrications and clean cites so a product cannot “study for” one exam at a time.
How scoring works
We do not publish one blended percentage. A single average hides the failure mode that matters. Instead we score six categories, each with its own bar. Timeouts and transport failures are excluded from each category’s analyzed count; publish analyzed and correct counts for Categories 1–5.
Category 1 — Overruled / reversed (bar: 100%)
These are real authorities that are no longer good law (or carry equivalent negative treatment). The product must confirm the identity and report that the case was overruled or similarly compromised. A silent “valid” is a fail. Lawyers who rely on that output will cite dead law with confidence.
Category 2 — Fabricated / identity traps (bar: 99.7%)
Fake cites, impossible locators, and wrong-identity traps (including real reporter pages paired with the wrong caption). The product must not hard-confirm. The bar is 99.7% rather than 100% because an adversarial suite can invent rare traps that almost never appear in real briefs; in practice, well-engineered checkers should clear this bar with room to spare.
Category 3 — Clean real cites (bar: 99%)
Proper cites to real cases that are not overruled for this test’s purposes. The product must not affirmatively call a good cite bad. False rejects waste attorney time and destroy trust.
Category 4 — Mangled, easier: Bluebook / form (bar: 95%)
Real cases written with basic Bluebook or form variation — abbreviation, spacing, court parenthetical, and similar noise that does not change the underlying authority. The right behavior is to confirm (or soft-confirm) when it is still the same case, optionally offering a preferred form. This is the everyday “almost Bluebook” problem, not OCR wreckage.
Category 5 — Mangled, harder: recovery problems (bar: 80%)
Real cases damaged in harder ways: wrong first page, transposed volume, OCR-style letter errors, structural breakage, and similar near-misses. Success means recovery: identify the intended case, flag the problem, and preferably suggest a correction. A bare “not in corpus” with no candidate is a miss even if declining was directionally cautious. This category is intentionally the hardest; many plausible resolutions exist, and there are endless ways to mangle a string.
Category 6 — Unresolved (unscored)
Inputs with no honest high-probability call — for example bare Id. with no antecedent. The product should say it cannot resolve the cite. There is no vanity percentage here.
For mangled Categories 4 and 5, “accept” on the answer key is not always enough by itself: harder rows require recovering the case, not merely declining a bad string.
CiteCheck results
| Cat | Name | Bar | Analyzed | Correct | Accuracy |
|---|---|---|---|---|---|
| 1 | Overruled / reversed | 100% | 187 | 187 | 100% |
| 2 | Fabricated | 99.7% | 2,452 | 2,452 | 100% |
| 3 | Clean real cites | 99% | 2,190 | 2,190 | 100% |
| 4 | Mangled — Bluebook / form (easier) | 95% | 71 | 71 | 100% |
| 5 | Mangled — hard problems | 80% | 349 | 301 | 86.2% |
| 6 | Unresolved | — | 47 | — | unscored |
CiteCheck clears every bar. Categories 1–4 are perfect on analyzed rows. Category 5, the recovery track for harder mangling, lands at 86.2% — above the 80% bar and well below the near-perfect scores on identity and Bluebook form, which is what you want from a difficult recovery exam: discrimination, not a vanity curve.
Headline identity performance on the full suite (excluding unresolved and a handful of unscored soft-lookup failures): CiteCheck hard-rejects zero graded rows, matches stance (confirm vs decline) on the vast majority of scored cites, and never hard-confirms a fabrication in Category 2 on the analyzed set.
Why split mangled cites in two
Treating every mangled string as one bucket hides an important product difference.
Bluebook and form variants are everyday errors. A checker that flinches at spacing or a missing court abbreviation will spam lawyers with false alarms. That track belongs at a high bar (95%+).
Harder mangling is a different job: reconstruct intent from a broken locator, show the candidate, and warn. That work is intrinsically messier. Scoring it at 80% keeps the bar honest while still demanding real recovery — not a shrug of “not found.”
If your vendor only quotes one mangled-cite percentage, ask which problem they measured.
How the key was built (without CiteCheck as the grade)
The answer key was locked with independent audits: corpus existence checks, structural second opinions, and stratified fresh review. CiteCheck is the product under test here; it was not used to write the grades it is scored against.
When a case resolves, the API is expected to report overruling / negative treatment as well as the identity verdict. Category 1 exists to make that requirement measurable.
Compare your checker
- Email results or questions: [email protected]
- Sign up · developer API ·
POST /api/v1/citecheck/cite - Public suite
Publish Category 1–5 rates (analyzed and correct), not one blended percentage. Include Category 4 and Category 5 separately so Bluebook form skill is not confused with hard recovery skill.