Key Takeaways
- • The Center for Democracy & Technology (CDT) submitted formal comments to NIST's Center for AI Standards and Innovation (CAISI) on draft guidance (AI 800-2) for automated benchmark evaluations of large language models, with a public comment period closing March 31, 2026.
- • NIST's draft guidance addresses critical evaluation methodologies including measurement construct definition, implementation protocols, and debugging procedures, while introducing "emerging practices" such as managing "evaluation awareness"—the phenomenon where models recognize they are being evaluated and potentially distort results.
- • CDT identified multiple areas for strengthening the guidance in future iterations, including enhanced clarity on evaluation as an iterative process, though the organization acknowledged the draft strikes "a thoughtful balance between conceptual foundations and practical implementation."
- • The guidance framework emphasizes alignment between evaluation goals and methods, encouraging developers to design assessments tailored to specific use cases rather than adopting popular approaches that may lack meaningful validity for their objectives.
- • At least 18 civil society organizations, including the Electronic Privacy Information Center (EPIC), have joined coalition efforts to provide feedback on the NIST AI 800-2 standard, reflecting broad stakeholder engagement in AI governance standards-setting.
Summary
The Center for Democracy & Technology submitted formal comments on March 25, 2026, responding to NIST's draft guidance for automated benchmark evaluations of large language models (NIST AI 800-2). The submission contributes to an open public comment period, which closes March 31, 2026, during which stakeholders can email feedback to [email protected].
NIST's Center for AI Standards and Innovation developed the draft guidance to establish best practices for how developers conduct robust automated evaluations of LLMs. The guidance covers measurement constructs, evaluation implementation, and troubleshooting processes. A notable element is the framework's treatment of "evaluation awareness"—a technical challenge where language models recognize they are being evaluated, potentially undermining the validity of benchmark results. The draft also incorporates emerging evaluation methodologies expected to gain importance as AI systems advance.
CDT acknowledged that the draft provides a balanced foundation combining conceptual rigor with practical applicability. The guidance encourages practitioners to clearly define evaluation objectives and align their measurement methodologies accordingly, discouraging reliance on default or convenient approaches that lack meaningful alignment with specific use cases. However, CDT's submission identifies areas requiring strengthening, particularly around conceptualizing evaluation as an iterative process rather than a one-time exercise.
The regulatory engagement reflects broader institutional interest in AI governance standards. EPIC joined 17 other civil society organizations in submitting coalition comments on the guidance, indicating coordinated civil society participation in shaping AI evaluation standards. The March 31, 2026 deadline represents a critical window for institutional stakeholders—including technology companies, investors, and compliance officers—to influence standards that will govern how AI systems are assessed and benchmarked across the industry.
NIST's Center for AI Standards and Innovation developed the draft guidance to establish best practices for how developers conduct robust automated evaluations of LLMs. The guidance covers measurement constructs, evaluation implementation, and troubleshooting processes. A notable element is the framework's treatment of "evaluation awareness"—a technical challenge where language models recognize they are being evaluated, potentially undermining the validity of benchmark results. The draft also incorporates emerging evaluation methodologies expected to gain importance as AI systems advance.
CDT acknowledged that the draft provides a balanced foundation combining conceptual rigor with practical applicability. The guidance encourages practitioners to clearly define evaluation objectives and align their measurement methodologies accordingly, discouraging reliance on default or convenient approaches that lack meaningful alignment with specific use cases. However, CDT's submission identifies areas requiring strengthening, particularly around conceptualizing evaluation as an iterative process rather than a one-time exercise.
The regulatory engagement reflects broader institutional interest in AI governance standards. EPIC joined 17 other civil society organizations in submitting coalition comments on the guidance, indicating coordinated civil society participation in shaping AI evaluation standards. The March 31, 2026 deadline represents a critical window for institutional stakeholders—including technology companies, investors, and compliance officers—to influence standards that will govern how AI systems are assessed and benchmarked across the industry.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.