Key Takeaways
- • The Center for AI Standards and Innovation (CAISI) at NIST has released a draft document, NIST AI 800-2, Practices for Automated Benchmark Evaluations of Language Models, and is requesting public comment through March 31, 2026.
- • NIST AI 800-2 sets out preliminary voluntary best practices for evaluating language models and AI agent systems, focused specifically on automated benchmark evaluations.
- • The draft is organized into three sections plus a glossary: defining evaluation objectives and selecting benchmarks, implementing and running evaluations, and analyzing and reporting results.
- • The primary audience is technical staff at AI deployers, developers, and third-party evaluators, though CAISI explicitly invites input from procurement specialists, business decision-makers, and technical integrators.
- • CAISI states that automated benchmarks cannot meet all AI evaluation objectives but are a common measurement tool, particularly useful where time, expertise, or resources are constrained.
Summary
The National Institute of Standards and Technology (NIST) published a draft document, NIST AI 800-2 Practices for Automated Benchmark Evaluations of Language Models, through its Center for AI Standards and Innovation (CAISI), and is soliciting public comment on it through March 31, 2026. According to the publication, consistent practices supporting the validity, transparency, and reproducibility of AI evaluations are only beginning to emerge, and the draft is intended to advance their development and voluntary adoption.
The document addresses automated benchmark evaluations, a specific measurement approach, and provides preliminary best practices for evaluating language models and AI agent systems. Its primary audience is technical staff at organizations evaluating AI systems, including deployers, developers, and third-party evaluators. NIST notes that all consumers of AI evaluation reports can benefit from well-communicated evaluation practices, which can inform AI procurement and implementation decisions. The agency states that such benchmarks cannot satisfy every evaluation objective, but are a common instrument where time, expertise, or resources are limited. CAISI anticipates issuing additional voluntary guidelines for other evaluation types.
The draft reflects CAISI's experience partnering with leading U.S. AI organizations to evaluate frontier AI models, along with ongoing measurement science research at NIST and elsewhere. It is organized into three sections and a glossary covering evaluation objectives and benchmark selection, implementation and execution, and analysis and reporting of results.
CAISI is seeking input on the usefulness and relative importance of the practices, missing practices within scope, which practices are emerging versus established, incorrect or unclear content, additional glossary terms, and when automated benchmarks are more or less useful than other evaluation paradigms. Feedback may be emailed to [email protected] in any form. CAISI does not plan to publish submissions, though correspondence may be subject to public disclosure.
The document addresses automated benchmark evaluations, a specific measurement approach, and provides preliminary best practices for evaluating language models and AI agent systems. Its primary audience is technical staff at organizations evaluating AI systems, including deployers, developers, and third-party evaluators. NIST notes that all consumers of AI evaluation reports can benefit from well-communicated evaluation practices, which can inform AI procurement and implementation decisions. The agency states that such benchmarks cannot satisfy every evaluation objective, but are a common instrument where time, expertise, or resources are limited. CAISI anticipates issuing additional voluntary guidelines for other evaluation types.
The draft reflects CAISI's experience partnering with leading U.S. AI organizations to evaluate frontier AI models, along with ongoing measurement science research at NIST and elsewhere. It is organized into three sections and a glossary covering evaluation objectives and benchmark selection, implementation and execution, and analysis and reporting of results.
CAISI is seeking input on the usefulness and relative importance of the practices, missing practices within scope, which practices are emerging versus established, incorrect or unclear content, additional glossary terms, and when automated benchmarks are more or less useful than other evaluation paradigms. Feedback may be emailed to [email protected] in any form. CAISI does not plan to publish submissions, though correspondence may be subject to public disclosure.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.