Key Takeaways
- • Full Fact's Public Interest AI Trust Benchmark found that leading chatbots gave reliable answers on many questions, but recurring weaknesses in timeliness, transparency, and consistency were identified across the models tested.
- • Full Fact tested five chatbots (Gemini 3.1, ChatGPT, Grok, DeepSeek and Claude) on 22 questions, collecting 7,625 responses between 15 July and 22 September 2026.
- • Chatbots sometimes presented outdated information as current, and several models gave single-answer responses on UK policy questions that omitted caveats specific to the UK's four nations.
- • DeepSeek was still identifying Keir Starmer as Prime Minister and Labour leader in September 2026, despite the change of Prime Minister on 20 July.
- • Full Fact says the findings are not a measure of how often errors occur, and that the report is the first in a series of assessments to be published over the next year.
Summary
Full Fact, the UK's independent fact-checking charity, has published the first report from its Public Interest AI Trust Benchmark, an independent assessment of the reliability of answers given by AI chatbots and the large language models behind them. The report argues that AI companies should not be marking their own homework, and it cites Microsoft AI's CEO, who told the BBC that there is broad agreement that AI companies should submit to independent auditors with access to their systems.
The benchmark scores chatbot answers against five criteria: factuality, transparency, timeliness, consistency and civic responsibility. Full Fact's fact checkers and data scientists manually annotated repeated responses to 22 questions, ranging from simple factual queries to contested civic topics such as voter ID and asylum support. The study ran from 15 July to 22 September 2026 and collected 7,625 responses from five chatbots, using APIs and allowing web searches. Full Fact reports that its atomic-fact extraction was accurate 97.8% of the time in a sampled check.
Full Fact describes the picture as mixed. Models gave reliable answers across many questions, and none repeated the core misinformation at the heart of the more sensitive civic questions. However, the report identifies recurring weaknesses. Chatbots sometimes presented outdated information as current or failed to recognise that a policy or figure had changed, which Full Fact says has real consequences when facts affect people's rights or entitlements. On questions where rules differ across the UK's four nations, several models gave broad single-answer responses that omitted necessary caveats. Some models were inconsistent in attributing claims to sources, and in a handful of cases one model abandoned a coherent answer altogether.
The report also notes that several models give longer answers than necessary, creating more opportunities for error. The change of Prime Minister on 20 July provided an unplanned test of how quickly models register changes in fact. Most models updated promptly, but DeepSeek was still identifying Sir Keir Starmer as both Prime Minister and Labour leader as late as September, and made similar errors about his age.
Full Fact stresses that the findings are drawn from specific recorded responses and are not a measure of how often such responses occur, and that the errors should not be read as a verdict on overall reliability. The organisation plans to publish a series of further assessments over the next year.
The benchmark scores chatbot answers against five criteria: factuality, transparency, timeliness, consistency and civic responsibility. Full Fact's fact checkers and data scientists manually annotated repeated responses to 22 questions, ranging from simple factual queries to contested civic topics such as voter ID and asylum support. The study ran from 15 July to 22 September 2026 and collected 7,625 responses from five chatbots, using APIs and allowing web searches. Full Fact reports that its atomic-fact extraction was accurate 97.8% of the time in a sampled check.
Full Fact describes the picture as mixed. Models gave reliable answers across many questions, and none repeated the core misinformation at the heart of the more sensitive civic questions. However, the report identifies recurring weaknesses. Chatbots sometimes presented outdated information as current or failed to recognise that a policy or figure had changed, which Full Fact says has real consequences when facts affect people's rights or entitlements. On questions where rules differ across the UK's four nations, several models gave broad single-answer responses that omitted necessary caveats. Some models were inconsistent in attributing claims to sources, and in a handful of cases one model abandoned a coherent answer altogether.
The report also notes that several models give longer answers than necessary, creating more opportunities for error. The change of Prime Minister on 20 July provided an unplanned test of how quickly models register changes in fact. Most models updated promptly, but DeepSeek was still identifying Sir Keir Starmer as both Prime Minister and Labour leader as late as September, and made similar errors about his age.
Full Fact stresses that the findings are drawn from specific recorded responses and are not a measure of how often such responses occur, and that the errors should not be read as a verdict on overall reliability. The organisation plans to publish a series of further assessments over the next year.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.