Your Cite Checker Is Probably Just Another LLM

Why a language model cannot verify citations — and what can

If an LLM drafted your motion, would you let the same LLM cite-check it?

Nobody says yes out loud. But that is what most “AI cite checking” on the market actually is: a second pass by the same kind of system, over the same document, reasoning from the same statistical picture of what a citation usually looks like. It is the witness authenticating their own testimony.

The problem is not that the models are bad at writing. The problem is what they are.

---

A language model is a probability engine, not a lookup

An LLM generates text one token at a time by sampling from a probability distribution. When it writes 410 U.S. 113, it is not retrieving a record. It is producing the digits that most plausibly follow Roe v. Wade, given everything it has read. When it later “reviews” that citation, it runs the same machinery again: does this string look like a citation that belongs to this case?

The answer it returns is a likelihood, dressed as a fact.

That distinction is invisible when the model is right — which is most of the time — and that is exactly what makes it dangerous. A tool that is right 97% of the time on citations still hands you a fabricated authority roughly once every few dozen cites. You will not know which one. Neither will the model. The wrong cite was generated by the same confident process as the right ones, and it carries the same confident tone.

Three properties follow from the architecture. No amount of prompting removes them.

It is not consistent. Run the same document through twice and you can get two different answers. Set temperature to zero and you have reduced the variance, not eliminated the cause — batching, kernel non-determinism, and silent model updates all move the output. A checker that grades your brief differently on Tuesday than it did on Monday is not a checker. It is a second opinion.

It is not independent. The verification signal comes from the same distribution that produced the error. If the model found Smith v. Jones, 412 F.3d 1102 (9th Cir. 2005) plausible enough to write, it will find it plausible enough to approve. Auditors are independent of the books for a reason.

It cannot prove a negative. A model has no way to distinguish “this case does not exist” from “I have not seen this case.” Ask whether an authority is real and you get a vibe, not a search. Absence of recall is not evidence of absence.

---

What cite-checking actually requires

Strip away the language problem and cite-checking is five discrete questions — none of them questions about language:

1. Existence — is there a decision at this volume, reporter, and page?
2. Identity — does that decision have the case name, court, and year the brief attributes to it?
3. Quotation — do the words inside the quotation marks appear in the opinion?
4. Treatment — has it been overruled, questioned, or superseded?
5. Form — is it in correct Bluebook form?

Every one of those is a lookup or a string comparison against a record. They are database operations. Handing them to a probabilistic text generator is a category error — like asking a very well-read colleague to recite a bank balance from memory instead of opening the account.

---

The failure mode that gets past everyone

An obviously fake citation is easy to catch. Zylor v. Bantham, 999 F.9d 1 fools nobody.

The one that gets filed is the real reporter cite bolted to the wrong caption. Smith v. Jones, 570 U.S. 744 (2013) — where 570 U.S. 744 is a genuine, published, still-good decision, and it is not Smith v. Jones. It survives a read-through because every component is individually correct. It survives an LLM review for the same reason: the pieces are all high-probability. Nothing about the string is anomalous. Only the join is wrong, and the join is a fact about a database, not a fact about language.

That shape has a name in LawDiver’s CiteChecker: name_mismatch. It is the signature of a hallucinated citation — and the specific thing the product exists to catch.

---

How LawDiver’s CiteChecker actually works

The architecture is the argument. LLMs are allowed to propose. Only code and the corpus are allowed to decide.

A reporter cite is a primary key. Volume + reporter + page — 570 U.S. 744 — is globally unique to a single decision. LawDiver canonicalizes it (570us744) and matches it exactly, never by substring or fuzzy containment, against the corpus. An exact key hit is not a high-confidence estimate. It is proof. That primitive is shared by the public API, the web portal, and the drafting app, so all three necessarily agree on what “valid” means.

The corpus is the authority. Roughly 10 million federal and state opinions, 18 million parallel-cite and reporter-key rows, 182 million extracted in-text citations, and 78 million resolved opinion-to-opinion citation edges — hundreds of gigabytes, harvested from public sources and continuously updated. The checker is not remembering the law. It is querying it.

Quotations are graded by alignment, not by judgment. Every quoted passage is checked against the actual source text with no model in the loop: offset-preserving normalization, n-gram anchoring to localize the passage, then Smith–Waterman local alignment to compare it word by word. Each quote comes back on a five-rung ladder — exact, almost exact (ellipses, [t]he, “cleaned up”), partial (a truncation), misquote (the wording was changed in the interior), or no match. Local alignment is what lets an honest truncation be told apart from an altered holding. Same document in, same grades out, every single time.

Good-law status is computed from the citation graph — 78 million real edges and their negative treatments — and “unknown” is reported separately from “negative,” so silence can never be mistaken for a clean bill of health.

The verdicts are disciplined, and honest about their limits. valid, name_mismatch, likely_valid, not_found, unverified, error. When the cite as written doesn’t key to anything, CiteChecker returns up to three candidates and refuses to pick one — silently “correcting” a citation to a case the author never read is the single worst thing a cite checker can do. When a source genuinely cannot prove absence (many statute sites return HTTP 200 for sections that do not exist), it says unverified and hands over the official link rather than making an accusation it can’t support.

Everything is auditable. Each run persists the citation as written, the authoritative Bluebook form, the source that settled it, the match quality, and the quote grade. You can hand that report to a partner, a client, or a judge.

Where do LLMs fit? One place: reading messy documents. Finding cite-shaped strings in a PDF, resolving id. and supra and short forms — a model is genuinely good at that, and CiteChecker uses one, harnessed and merged with deterministic extraction. But an extractor’s job is to say here is a string that might be a citation. It never gets to say and it’s fine. The database says that.

---

The test

Run your cite checker over the same brief twice.

If the two reports differ, you don’t have a cite checker — you have a colleague with a good memory and no library card. Under Rule 11 and its state analogues, “the AI said it was fine” is not a defense, and the sanctions docket now has enough entries to prove it.

Your drafting tool can be probabilistic. Your verification tool cannot be.

Try CiteDiver — upload a brief and see every citation verified against real authority.