Key Takeaways
- • A PNAS paper (received May 31, 2025; accepted October 1, 2025; published July 20, 2026) argues that legal AI needs public benchmarking comparable to that used in medicine, software engineering, and facial recognition, because commercial legal AI systems are largely illegible to the public.
- • The authors identify six design dimensions of benchmarking, including task definition, benchmark construction, and evaluation protocol, that determine what information a benchmark produces and how it can be captured, watered down, or abused.
- • The paper cites an academic study finding a Thomson Reuters AI legal research tool hallucinated in 33% of benchmark queries, versus a 10% internal error rate the company reported, attributing the gap to differences in benchmark construction.
- • The authors propose matching institutional models to resources: NIST's Face Recognition Vendor Test for high-resource government coordination, industry consortia for medium resources, and academic grants for low-resource targeted research.
- • The paper cites more than 120 documented cases of AI-fabricated citations in legal filings, underscoring the stakes of unverified performance claims.
Summary
A PNAS paper titled "There is no free benchmark: An institutional view of legal AI benchmarking" argues that the legal sector needs public AI benchmarking on par with that of other domains. The authors contend that legal AI suffers from "illegibility," meaning there is little public information about the design and performance of many commercial legal AI systems. The paper notes that the law has become a showcase for generative AI potential, with a flurry of new tools for contract review, legal research, litigation, compliance monitoring, and e-discovery following the release of ChatGPT. At the same time, the authors cite documented hallucinations, bias, and uneven performance across areas of law, along with more than 120 documented cases involving AI-fabricated citations.
The paper frames benchmarking as a discretionary, institutionally shaped process with six design dimensions. These include task definition, where narrow benchmarks may overlook vital functions or mislead by highlighting favorable capabilities, and benchmark construction, where data selection can shift difficulty and external validity. The authors illustrate these risks with two examples. An academic study found that a Thomson Reuters AI legal research tool hallucinated in 33% of benchmark queries, while Thomson Reuters reported an internal error rate of 10%; the authors attribute the discrepancy to differing benchmark construction. Separately, Paxton AI's press release claiming 94%+ accuracy on a Stanford hallucination benchmark was described by the authors as relying on a dataset so easy that simple bag-of-words models achieve near-perfect performance.
The authors argue that an institutional lens, focused on benchmark ownership, incentives, and capacity, is needed as legal AI moves from academic research to deployed systems. They identify stakeholders including developers, purchasers, academics, bar associations, courts, and governmental oversight bodies. Rather than advocating a single best approach, the paper matches three institutional models to resource levels: NIST's Face Recognition Vendor Test for high-resource government coordination, industry consortia for medium-resource private leadership, and discrete academic grants for low-resource targeted research.
The paper frames benchmarking as a discretionary, institutionally shaped process with six design dimensions. These include task definition, where narrow benchmarks may overlook vital functions or mislead by highlighting favorable capabilities, and benchmark construction, where data selection can shift difficulty and external validity. The authors illustrate these risks with two examples. An academic study found that a Thomson Reuters AI legal research tool hallucinated in 33% of benchmark queries, while Thomson Reuters reported an internal error rate of 10%; the authors attribute the discrepancy to differing benchmark construction. Separately, Paxton AI's press release claiming 94%+ accuracy on a Stanford hallucination benchmark was described by the authors as relying on a dataset so easy that simple bag-of-words models achieve near-perfect performance.
The authors argue that an institutional lens, focused on benchmark ownership, incentives, and capacity, is needed as legal AI moves from academic research to deployed systems. They identify stakeholders including developers, purchasers, academics, bar associations, courts, and governmental oversight bodies. Rather than advocating a single best approach, the paper matches three institutional models to resource levels: NIST's Face Recognition Vendor Test for high-resource government coordination, industry consortia for medium-resource private leadership, and discrete academic grants for low-resource targeted research.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.