Key Takeaways
- • Harvey released v1.1 of LAB-AA, its legal agent benchmark run by Artificial Analysis, adding a hallucination check that tests whether each model claim is explicitly supported by the underlying source documents.
- • Top models on the LAB-AA leaderboard post criterion pass rates of roughly 93-95% as of October 2026, while all-pass rates, which require satisfying every criterion on a task, often fall below 30%.
- • The v1.1 update layers source-grounding scores onto the existing expert-rubric scoring, targeting answers that satisfy rubric criteria while padding with unsupported claims.
- • LAB-AA uses a private set of 120 tasks, which prevents models from being tuned to the test set and keeps the leaderboard more reliable.
Summary
Harvey has released version 1.1 of LAB-AA, the legal agent benchmark run by the independent AI evaluation firm Artificial Analysis and tied to Harvey's open-source Legal Agent Benchmark (LAB), which Harvey released on May 6, 2026. The update adds hallucination checking to how models are scored. Outputs are now judged on whether their claims have explicit support in the source documents, rather than only on whether they satisfy expert-written rubric criteria.
LAB-AA launched around July 7, 2026, with a private set of 120 tasks. Keeping the tasks private is intended to prevent models from being tuned to questions they have already seen. The original article describes a single-judge scoring system. A separate Harvey announcement of v1.1 describes a three-judge panel grading every rubric criterion, so the final grading architecture should be confirmed against the official methodology before it is relied on for comparisons.
According to the report, top models on the leaderboard post criterion pass rates of roughly 93-95% as of October 2026. All-pass rates, which require a model to satisfy every criterion on a task, often sit below 30%. The benchmark's designers argue this gap matters because legal work has binary consequences: a brief that succeeds on nine arguments but fails on the tenth can still lose the motion.
The hallucination check addresses a specific weakness in rubric-only scoring. A model could satisfy many rubric criteria while adding claims the documents do not support. Under v1.1, such unsupported claims are scored against the source material, which the benchmark's developers say is designed to catch exactly that behavior.
LAB-AA launched around July 7, 2026, with a private set of 120 tasks. Keeping the tasks private is intended to prevent models from being tuned to questions they have already seen. The original article describes a single-judge scoring system. A separate Harvey announcement of v1.1 describes a three-judge panel grading every rubric criterion, so the final grading architecture should be confirmed against the official methodology before it is relied on for comparisons.
According to the report, top models on the leaderboard post criterion pass rates of roughly 93-95% as of October 2026. All-pass rates, which require a model to satisfy every criterion on a task, often sit below 30%. The benchmark's designers argue this gap matters because legal work has binary consequences: a brief that succeeds on nine arguments but fails on the tenth can still lose the motion.
The hallucination check addresses a specific weakness in rubric-only scoring. A model could satisfy many rubric criteria while adding claims the documents do not support. Under v1.1, such unsupported claims are scored against the source material, which the benchmark's developers say is designed to catch exactly that behavior.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.