How Was the LawDiver Caselaw Database Assembled?

· LawDiver

From Harvard CAP and CourtListener bulk dumps to daily harvest, EyeDiver, the Bubbler, embeddings, dual-LLM analysis, and GoodLaw — how LawDiver built a free-to-use American caselaw corpus without Westlaw or Lexis content.

How Was the LawDiver Caselaw Database Assembled?

Short answer: We started with open, unrestricted bulk corpora (Harvard’s Caselaw Access Project and CourtListener), reconciled them against each other, then ran a continuous harvest across federal and state court sites. Every opinion is identified, scanned for citations by EyeDiver, cross-indexed by the Bubbler, embedded for semantic search, and analyzed by a dual-LLM pipeline that powers GoodLaw. No Westlaw. No Lexis. No Shepard’s content.

This is the account of where the data came from and how we process it. If you want wholesale access, see Bulk Data.

On this page

  1. Why we publish this
  2. Layer one: Harvard Caselaw Access Project
  3. Layer two: CourtListener quarterly bulk data
  4. Reconciliation: CAP vs CourtListener
  5. Where people report CourtListener data problems
  6. Layer three: continuous harvest
  7. Juriscraper, modified for LawDiver
  8. From PDFs to text
  9. Identity: how a case becomes a row
  10. EyeDiver: finding every authority in the text
  11. The Bubbler: matching cites to authorities
  12. When the database repairs itself
  13. Embeddings and semantic search
  14. Dual-LLM analysis and GoodLaw
  15. Humans in the loop
  16. Copyright, reporters, and why the moat is collapsing
  17. Scale today
  18. A billion-dollar question
  19. Frequently asked questions
  20. Sources
  21. Related reading

Why we publish this

When we say LawDiver’s caselaw data is free from copyright or other contractual restrictions that would block ordinary research use, it is not enough to assert it. We have to show the pipeline.

First, provenance matters. Judicial opinions are public. The commercial scaffolding around them — headnotes, Key Numbers, proprietary citators — is not what we ingest. Saying “public domain / unrestricted bulk sources + our own harvest + our own analysis” is only credible if we can name the sources and the steps.

Second, if we will not explain how we process the data, nobody should believe us when we claim there are no vendor lock-in terms sitting on top of it. This write-up is evidence of technical competence: we did what we claim we did.

Third, moats around unprotectable data such as caselaw text are collapsing. It is possible to “build your own.” It is also incredibly hard, because the data is heterogeneous: fifty state systems, federal courts, PDF scans, mangled OCR, incomplete reporter keys, duplicate clusters, and citation forms that lawyers invent faster than parsers can keep up. We publish this so readers understand both halves of that sentence.

Layer one: Harvard Caselaw Access Project

We began with the Caselaw Access Project (CAP), Harvard Law School Library’s digitization of roughly 40 million pages of U.S. court decisions into a structured dataset of about 6.7 million cases spanning roughly 360 years of American legal history (volumes continuously added through about 2020).

Harvard and partners made that corpus available for open use. A convenient machine-learning-ready mirror lives on Hugging Face as free-law/Caselaw_Access_Project (see also Harvard’s bulk service at case.law and Free Law Project’s documentation). That dump is the historical backbone: state and federal reporters that no live crawler could rebuild from court websites alone.

CAP is extraordinary. It is also incomplete for a modern research product: coverage thins after the digitization window, some metadata is wrong, and duplicate or mislabeled citations appear often enough to corrupt downstream graphs if you trust every row.

Layer two: CourtListener quarterly bulk data

Next we downloaded the latest quarterly bulk snapshots from CourtListener / Free Law Project. Those files are regenerated on the last day of March, June, September, and December and published as full-table CSV snapshots (not incremental deltas) in a public S3 listing:

https://com-courtlistener-storage.s3-us-west-2.amazonaws.com/list.html?prefix=bulk-data/

Opinion clusters, opinions (full text), dockets, courts, and citation maps bring the historical corpus current through the end of the quarter. Free Law Project states that the bulk files are free of known copyright restrictions.

CourtListener also scrapes and partners for post-CAP opinions. Together with CAP, it is the largest open American caselaw foundation available without a commercial research license.

Reconciliation: CAP vs CourtListener

Open does not mean clean.

We compared CAP rows to CourtListener bulk rows — updating texts, correcting metadata, and resolving conflicts where the two sources disagreed. Both corpora are provided without copyright or contractual restrictions that would prevent this kind of reconciliation. Both still have:

  • Missing or wrong reporter citations
  • Duplicate opinion / cluster issues that can corrupt citation graphs and search results
  • Gaps in recent Southern, Atlantic, and other regional reporter keys that lawyers still must file

Reconciliation is not optional polish. It is the difference between a dump and a database.

Where people report CourtListener data problems

CourtListener is open about the fact that a corpus this large has gaps — and the community has a public place to talk about them.

Free Law Project’s docs point questions about the data to GitHub Discussions on the CourtListener repo — that is the live forum. Researchers and builders post missing opinions, empty clusters, docket-number lookup failures, RECAP-vs-opinions mismatches, bulk-export integrity quirks, and API gotchas. Concrete bugs also land as GitHub Issues (often tagged data quality): for example, opinions that exist in RECAP but not in the opinions index, or citation-map rows that point at deleted opinion ids after a quarterly export race.

Separately, Free Law Project’s own blog at free.law covers product and scanning work (including how court-site scrapers get text before reporter pagination exists). The blog is news and method; the Discussions + Issues boards are where people file “this case is wrong / missing / unsearchable.”

That public trail matters for anyone assembling a corpus: the open dumps are invaluable, and the open issue log is proof that “download once and trust forever” is not a strategy. LawDiver’s Bubbler and harvest lanes exist in part because those failure modes are real and well documented.

Layer three: continuous harvest

Bulk dumps lag. Law moves every business day.

So we run a massive caselaw harvesting operation across the states and the federal system.

Federal caselaw

Federal opinion text and complementary primary law arrive through public government channels — principally GovInfo (GPO) for federal opinions, the United States Code, and the C.F.R. — plus court-site publication and public RSS/ECF signals. LawDiver does not subscribe to a commercial caselaw feed and does not scrape Westlaw or Lexis for opinion text or headnotes.

State caselaw

For state law we maintain a Direct Harvest registry of in excess of 200 state and territorial court targets (supreme courts, intermediate appellate courts, and related publishers), with federal courts on additional lanes — well more than 200 court websites in the continuous harvest surface overall. Harvesters run daily (and, for many courts, around the clock): HTML listings, verified RSS/Atom/JSON feeds when a court exposes them, and fallback scrapers when the site only offers a human-facing docket page.

Courts do not publish on one schedule. Some post at dawn, some after close of business, some in weekly batches. Continuous harvest is how yesterday’s opinion is in the index today rather than in next quarter’s dump.

Juriscraper, modified for LawDiver

We also use the open-source Juriscraper library from Free Law Project — and we heavily adapted it for LawDiver’s production harvest lane.

What that means in practice:

  1. Sidecar service, not a laptop script. Juriscraper runs as an internal FastAPI service (juriscraper-svc) that the LawDiver worker calls over the private network. Health checks, module listing, and scrape endpoints are first-class ops surfaces — not ad-hoc CLI invocations.
  2. Schema normalization. Raw Juriscraper case objects are mapped into LawDiver’s opinion / cluster / citation identity model (court id, filing date, docket, party names, download URLs) before anything is persisted.
  3. Layered with native sources. Where a court exposes a reliable RSS, Atom, or JSON feed, we prefer that verified feed. Juriscraper fills gaps and covers courts whose HTML would otherwise require fragile one-off parsers.
  4. Breadth across the U.S. tree. The sidecar loads opinion modules under federal appellate, district, special, bankruptcy, state, and territories roots — then LawDiver’s harvest registry decides which modules are live for which courts on which day.
  5. Court-site churn handling. Publisher HTML breaks constantly. Our harvest ops stack (dry-run windows, zero-yield monitoring, per-court enablement) treats Juriscraper as an instrumented dependency, not a black box.

Open source got us 80% of the scraper surface. Production reliability is the other 20% — and it is most of the work.

From PDFs to text

Much of modern caselaw arrives as PDFs. Some have a clean text layer. Many are scans.

LawDiver’s extraction pipeline is a stack of open and commodity tools, not a single magic box:

  1. Native PDF text via pdf.js / pdf-parse (and related layout-aware extraction) when the file already contains selectable text.
  2. Rasterization of pages when the text layer is missing or garbage.
  3. Tiered OCR:
  • Tesseract first (fast, local),
  • image preprocessing and a second Tesseract pass when quality gates fail,
  • PaddleOCR as a sidecar when enabled,
  • AWS Textract as the last resort for stubborn scans.
  1. Quality gates (characters/words per page, confidence thresholds) so bad OCR does not silently poison the corpus.
  2. Gotenberg in the stack for PDF rendering / conversion jobs where the product needs a reliable print path.

The ability of ordinary citizens and small teams to take open-source tools from GitHub, install them on a cloud server, and have the thing actually work is — at most — about a decade to twelve years old in its modern form (Docker 1.0 and Docker Hub arrived in 2014; GitHub Actions CI/CD for public repos matured around 2019). Before that, “clone the repo and run it in production” was a specialist craft. Today it is still hard — but it is no longer impossible without a Fortune 500 IT department.

Identity: how a case becomes a row

Every case in LawDiver receives a unique identifier, then a stable identity bundle:

  • Citations (reporter keys and parallels, when known)
  • Case / docket number
  • Decision date
  • Jurisdiction / court

That identity is what search, cite-checking, and the citation graph hang on. Without it, you have a pile of PDFs. With it, you have a corpus.

Duplicates are actively hunted. CAP and CourtListener both contain near-duplicate opinions; harvest can re-fetch the same slip. Deduplication and cluster hygiene are continuous jobs, not a one-time import script.

EyeDiver: finding every authority in the text

How do we match citations in cases to the cases (and statutes, and rulings) themselves?

We built a proprietary scanner — EyeDiver — on top of the open-source eyecite project from Free Law Project, then greatly expanded it for LawDiver’s authority surface.

EyeDiver walks opinion text (and other documents in the system) and finds:

FamilyExamples
Caselaw531 U.S. 98, 799 F.2d 1219, 772 So. 2d 1243, short forms, id., supra
StatutesU.S.C., state codes, session laws
RegulationsC.F.R., Treasury Regulations, state administrative codes
Court rulesFRCP, FRE, statewide rules of procedure and evidence
Tax authoritiesRevenue Rulings (Rev. Rul.), Revenue Procedures (Rev. Proc.), IRS Notices, Private Letter Rulings (PLRs), Technical Advice Memoranda (TAMs), Treasury Decisions
Immigration authoritiesBoard of Immigration Appeals decisions (often Matter of [Name]), Immigration & Nationality Decisions (I&N Dec.), Administrative Appeals Office (AAO) decisions, BALCA (Board of Alien Labor Certification Appeals) decisions, and related EOIR / DHS guidance when cited as authority
SEC authoritiesSEC ALJ initial decisions (File No. 3-XXXXX), Commission opinions/orders, and adjudicatory releases (Exchange Act Release No. 34-…, Advisers / Investment Company Act releases)
EPA EAB / E.A.D.Environmental Appeals Board decisions published in Environmental Administrative Decisions (E.A.D.), plus related EAB slip and unpublished orders when cited as authority
NLRB authoritiesNational Labor Relations Board Board decisions (328 N.L.R.B. 433, 375 NLRB No. 38) and ALJ decisions (JD-61-26, JD(SF)-16-26)
USPTO board authoritiesPTAB / TTAB / BPAI proceeding numbers, U.S.P.Q. parallels, and Ex parte / In re board parentheticals
Patent / trademark board / ITCPTAB proceeding ids (IPR2016-01234, PGR…, CBM…), USPQ / USPQ2d / USPQ3d, (P.T.A.B.) / (T.T.A.B.) / (B.P.A.I.) parentheticals, and ITC § 337 investigation numbers (Inv. No. 337-TA-1234, bare 337-TA-1234)

EyeDiver has been run across the corpus at scale — on the order of 10 million completed scans, producing close to 200 million in-text citation spans (with character offsets and pin cites) and close to 100 million resolved opinion-to-opinion edges.

Deterministic scanning is the backbone: same text in, same spans out. That is what makes a citator graph rebuildable and auditable.

The Bubbler: matching cites to authorities

EyeDiver finds strings. The Bubbler is LawDiver’s cross-index: every authority EyeDiver identifies is matched, wherever possible, to a known authority already in the system — a case cluster, a statute section, a tax ruling, an immigration decision.

Think of it as the join between “what the opinion wrote” and “what exists in the database.”

When EyeDiver extracts 410 U.S. 113, the Bubbler resolves that reporter key to Roe v. Wade (or reports that it cannot). When the string is a Rev. Rul. or an I&N Dec. cite, the Bubbler aims at those authority tables the same way.

Unresolved cites are not trash. They are signals — often signals that our row for the cited case is missing a reporter key, or that the citing opinion mistyped a volume, or that a parallel cite was never attached upstream.

When the database repairs itself

Open-source caselaw files from Harvard and CourtListener are incomplete, and in places wrong, on case citations. They also carry duplicate caselaw that can corrupt outputs if left alone.

So every case that runs through EyeDiver and the Bubbler participates in a repair loop:

  1. Cite inside a scanned opinion, no match in our index — often because the target case in our database is missing that citation. High-confidence proposals use the cite as it appears in the citing opinion to create or fix the citation on the target case.
  2. Cite accuracy checks — Is the cite in the new opinion correct? Does it match an existing case? Is the citation already stored on that case wrong?
  3. Confidence filter — The Bubbler surfaces correction opportunities, scores them, and writes repairs only when confidence is high enough. Low-confidence candidates stay in review queues rather than silently rewriting history.

Example: New York reporter-series mislabels

New York’s citation system is notoriously difficult. We saw systemic problems: a large group of New York Court of Appeals cases inherited from the Harvard project carried canonical N.Y. citations that should have been N.Y.2d (wrong reporter series for the era). Fault detection flagged the pattern; we implemented a global repair that demotes the bad first-series form and attaches the correct second-series (or third-series) cite — while keeping the incorrect form as a non-canonical alias so old strings still resolve.

That is the difference between “we imported a dump” and “we operate a corpus.”

Example: Florida So. 3d citations hiding in plain sight

Sometime in the mid-2010s into the 2020s, Southern Reporter (So. 3d) assignment metadata for many Florida opinions stopped flowing reliably into free bulk sources. CourtListener often lacks the key. The data is not really scrapable off a convenient public page.

But you know where the data still lives?

Inside the cases already in our database.

Later opinions cite earlier Florida decisions with the So. 3d string. The Bubbler finds those cites, matches them to Florida opinions that lacked a Southern key, and — when confidence clears the bar — attaches the reporter citation to the proper case. We have run that reconstruction across large So. 3d volume ranges precisely because Rule 9.800 still wants the Southern cite even when free metadata does not provide it. (Deep dive: Why caselaw citations still depend on print reporters.)

West has no copyright on the bare citation system as such, and — under the leading modern appellate treatment — very likely no copyright on star pagination either. In Matthew Bender & Co. v. West Publishing Co., 158 F.3d 693 (2d Cir. 1998), the Second Circuit held that inserting West page numbers into an independent compilation of opinions does not infringe West’s compilation copyright; page breaks are unprotected facts under Feist Publications, Inc. v. Rural Telephone Service Co., 499 U.S. 340 (1991). (An earlier Eighth Circuit decision, West Publishing Co. v. Mead Data Central, Inc., 799 F.2d 1219 (8th Cir. 1986), had gone the other way on star pagination; Feist and Matthew Bender are the modern doctrinal floor.)

Copyright doctrine does not force West to hand you a tidy API. Practical difficulty is not the same thing as legal exclusivity. Big Data plus cheap cloud plus modern codegen closes that gap for teams that can do the engineering.

Embeddings and semantic search

Every case is run through an embedding LLM so the corpus supports semantic search.

Plain-English definitions:

TermMeaning
LLMLarge language model — a neural network trained on vast text that can read, summarize, classify, and generate language.
EmbeddingA list of numbers (a vector) that represents the meaning of a passage. Similar legal ideas land near each other in vector space even when the words differ (“unlawful detainer” ≈ “eviction for nonpayment”).
Embedding LLM / embedding modelA model specialized to produce those vectors (LawDiver uses Qwen3-Embedding-8B at 1024 dimensions).
Semantic searchRetrieval by meaning (vector nearest neighbors), not only by exact keywords.
Cosine similarityA common way to score two vectors by the angle between them: same direction ≈ strong match.
Boolean / keyword searchExact constraints — reporter cite, statute section, date, court, quoted phrase. LawDiver runs both semantic and boolean retrieval.

We store vectors in PostgreSQL with pgvector (half-precision) and retrieve with a fast binary HNSW first pass plus rerank. Two vector sets matter: one for opinion body text, one for LawDiver’s own analysis card (issue, holding, posture). A question phrased like a lawyer’s issue often hits the analysis set; the body set catches what analysis omitted.

Dual-LLM analysis and GoodLaw

After identity and citation scanning, cases run through two LLMs (a dual pass) to:

  • Analyze citations in context
  • Summarize cases
  • Classify cases
  • Extract holdings

The most important job of that dual-LLM process is treatment: what does this case do with the authorities it cites?

  • Does it overrule an older case? Mark the older case accordingly.
  • Does it criticize, question, or distinguish it?
  • Does it follow, explain, or cite in passing?

Users need that graph. A cite without treatment is a string. A cite with treatment is research.

That pipeline is the foundation of GoodLaw — LawDiver’s own citator signal for whether a case remains good law. Our target operating claim is 99.9% accuracy on detecting overruled status for cases our systems have fully processed, with continuous improvement as coverage and models advance. GoodLaw is not Shepard’s and not KeyCite. It is built from LawDiver’s corpus, EyeDiver spans, Bubbler edges, and LLM treatment analysis — not from a licensed extract of a commercial citator.

Always-on web sweeps

We also run web scrapers 24/7 that revisit authorities and look for something we did not know yesterday: Has a case been overruled or vacated recently in a source our harvest lane has not yet absorbed? External signals feed the same GoodLaw and human-review queues so the database stays as current and accurate as possible.

Humans in the loop

Automation is the default. Significant changes are not left on autopilot.

When a major data change is proposed — especially classifying a case as overruled when automated systems missed it, or a bulk citation repair that would rewrite thousands of keys — humans review before the write becomes canonical. Confidence thresholds, review queues, and reversible origins on citation rows exist so a bad automatic “fix” can be audited and undone.

That is how you get scale without pretending machines are infallible.

Copyright, reporters, and why the moat is collapsing

Judicial opinions are not West’s copyrighted novels. The hard problems are metadata, citation infrastructure, OCR, deduplication, and treatment analysis.

West can still make reporter citations operationally scarce. Paying a commercial research vendor remains the fastest path for many firms. But the power of modern technology — enormous databases at trivial unit cost, scalable cloud compute, and the ability to generate specialized code for a fraction of last year’s cost, often in minutes — gives Big Data leverage to people who could not access it a year ago.

What was impossible for ordinary citizens and small businesses twelve months ago is now possible. It still requires expertise in law, database design, programming, and cloud infrastructure. The moat did not vanish because the problem got easy. The moat is collapsing because the tools got cheap enough for small teams with the right skills.

Scale today

LawDiver’s live store is not a single SQLite file on a laptop. It currently holds well over 110 billion bits of structured and textual information, spread across:

  • PostgreSQL 18 (control, corpus, and search databases)
  • OpenSearch 3.x indices for full-text and related retrieval (text and vector search surfaces)

Public, rounded corpus facts we stand behind on the marketing site include on the order of 10 million opinions, close to 100 million citation edges, close to 200 million in-text citations, and hundreds of gigabytes of corpus storage — with continuous harvest adding more every day. Measured engineering detail lives on Bulk Data.

The software that makes those elements work — harvest, identity, EyeDiver, the Bubbler, embeddings, dual-LLM analysis, GoodLaw, and the product surfaces around them — covers about 525,000 lines of code. That is an engineering feat that would have taken a team of engineers roughly six months if this project had been built in 2025 or earlier. New coding technology makes possible what would have been, less than a year ago, impossible or cost-prohibitive without a large funding round.

A billion-dollar question

When Clio completed its US$1 billion acquisition of vLex (deal announced June 30, 2025; closed November 10, 2025 — widely reported as the largest M&A transaction in legal technology), we thought: wow — is a comprehensive American caselaw database really worth a billion dollars?

Well, we hope so.

Anyone who wants to buy LawDiver’s database is welcome to contact us. We will give you a great deal — much better than the vLex deal.

Frequently asked questions

Is LawDiver’s caselaw copied from Westlaw or Lexis?

No. The historical backbone is Harvard CAP and CourtListener bulk data (both unrestricted for this use). Currency comes from public court sites, GovInfo, feeds, and our own harvest. Analysis, GoodLaw, and embeddings are LawDiver’s own work product — not West headnotes, not Lexis headnotes, not Shepard’s.

What is EyeDiver?

EyeDiver is LawDiver’s citation scanner. It started from Free Law Project’s open-source eyecite and was expanded in-house to cover a wider authority set (including tax, immigration, SEC, EPA EAB, NLRB Board/ALJ, and USPTO board authorities) and to feed LawDiver’s reporter-key index at corpus scale.

What is the Bubbler?

The Bubbler is the cross-index that matches EyeDiver’s extracted citations to authorities in the database — and that proposes high-confidence citation repairs when citing opinions reveal missing or wrong keys on target cases.

What is GoodLaw?

GoodLaw is LawDiver’s citator layer: treatment analysis (overrule, distinguish, criticize, and related signals) derived from our corpus and dual-LLM pipeline, plus continuous sweeps for newly negative authority. It is not a licensed Shepard’s or KeyCite feed.

Why do Florida So. 3d cites matter so much?

Because Fla. R. App. P. 9.800 still expects Southern Reporter citations for ordinary Florida appellate opinions. A database with the PDF but without the So. 3d string is incomplete for Florida filing practice. See our deeper article on print reporters and citation infrastructure.

Can I build my own caselaw database?

Yes — in principle. CAP and CourtListener give you a starting mountain of public text. Building a reliable research corpus still means harvest engineering, PDF/OCR pipelines, citation repair, deduplication, embeddings, treatment analysis, and ops discipline. Possible. Not casual.

Where can I get LawDiver data wholesale?

See Bulk Data, or contact us about a corpus license. If you were comparing notes with a billion-dollar acquisition multiple, bring coffee — we are friendlier on price.

Sources

Related reading

How LawDiver Built Its Caselaw Database | Sources & Methods | LawDiver