Key Takeaways
- • Epoch AI's benchmarking reports that frontier AI models score up to 65% on its 11-task automation evaluation, per the headline of the Epoch report covered by TOKENPOST.
- • The evaluation found AI systems handled structured work better than open-ended assignments, indicating that current frontier models remain limited in automating complex tasks.
- • Epoch AI's publisher materials state that open-weight Chinese models trail the overall frontier by about seven months on FrontierMath Tiers 1-3.
- • Epoch's data places the ECI frontier advancing about 14 points per year since the introduction of reasoning models, a trend cited in its publisher materials.
Summary
Epoch AI's latest automation research, as reported by TOKENPOST, finds that leading frontier AI models score up to 65% on an 11-task evaluation designed to test automated work. The report's headline finding, titled "Epoch's Automation Reports Find Frontier AI Still Can't Do...," indicates that the systems performed better on structured work than on assignments requiring less defined task structures. The full article text was not available for this summary, so the methodology, individual model scores, and the specific task categories could not be verified.
Epoch AI's broader publisher materials provide context on the capabilities and benchmarking landscape. According to Epoch, the ECI frontier has advanced by about 14 points per year since the introduction of reasoning models, and AI chip performance per dollar has grown by 49% per year. On the company's FrontierMath benchmark, a set of hundreds of unpublished, highly difficult mathematics problems, Epoch reports that open-weight Chinese models' top scores on Tiers 1-3 lag the overall frontier by roughly seven months. Epoch also reports that model scores on its IKEA furniture assembly benchmark rose from 28% to 80% over ten months, with open-weight models trailing closed-weight models. These figures come from Epoch's undated publisher pages and are not confirmed as part of the 65% automation finding.
Epoch maintains a registry of 89 AI benchmarks spanning mathematics, software engineering, agentic workflows, and games. Its materials say the benchmarks are used to track model performance across these domains, and they are the basis for the company's comparative capability assessments.
## DATA
Epoch AI's broader publisher materials provide context on the capabilities and benchmarking landscape. According to Epoch, the ECI frontier has advanced by about 14 points per year since the introduction of reasoning models, and AI chip performance per dollar has grown by 49% per year. On the company's FrontierMath benchmark, a set of hundreds of unpublished, highly difficult mathematics problems, Epoch reports that open-weight Chinese models' top scores on Tiers 1-3 lag the overall frontier by roughly seven months. Epoch also reports that model scores on its IKEA furniture assembly benchmark rose from 28% to 80% over ten months, with open-weight models trailing closed-weight models. These figures come from Epoch's undated publisher pages and are not confirmed as part of the 65% automation finding.
Epoch maintains a registry of 89 AI benchmarks spanning mathematics, software engineering, agentic workflows, and games. Its materials say the benchmarks are used to track model performance across these domains, and they are the basis for the company's comparative capability assessments.
## DATA
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.