Key Takeaways
- • Google Cloud AI Research released PLE: Physical-Engineering's Last Exam, a benchmark of 85 long-horizon engineering tasks, and the top agent, Anthropic's Claude Opus 5.5, posted a mean score of 50.6%.
- • Claude Opus 5.5 met every hard engineering constraint in only 41.3% of design tasks, the metric the research treats as the more telling result.
- • OpenAI's GPT-6 Astra scored 48.3%, while Google's own Gemini configurations landed in the low-to-mid 40s.
- • Claude Opus 5.5 consumed roughly 27 million tokens and 63 minutes per task at approximately $17 in inference cost per task.
- • The benchmark keeps 56 of its 85 tasks hidden for a potential competition leaderboard, with 29 development tasks released publicly.
Summary
Google Cloud AI Research has released PLE: Physical-Engineering's Last Exam, a benchmark designed to test whether frontier AI agents can perform work that engineers bill for, according to the article published by Crypto Briefing. The benchmark contains 85 long-horizon tasks across four disciplines: 30 mechanical, 27 electrical, 18 civil and 10 structural. Each task requires a billable deliverable such as a calculation package or design review, and agents work from a project corpus that averages 61 files per task.
Claude Opus 5.5, developed by Anthropic, led the field with a mean score of 50.6%, averaged across three runs. OpenAI's GPT-6 Astra followed at 48.3%, and Google's own Gemini configurations scored in the low-to-mid 40s. The researchers emphasized the constraint-pass figure: the top agent satisfied every hard engineering constraint in only 41.3% of design tasks. The article notes that these constraints govern outcomes such as structural integrity and electrical safety, and that a mean score can reward partial credit even when a design violates a non-negotiable requirement.
The top performance carried a substantial compute cost. Claude Opus 5.5 consumed roughly 27 million tokens and needed 63 minutes per task, equating to approximately $17 in inference costs per task. The research framing states that meeting stringent constraints is where the difficulty lies and that even leading models have substantial room to improve.
The paper appeared on arXiv around October 21-23, 2026, under identifier arXiv:2610.30537. Google published a public GitHub repository containing 29 development tasks and supporting tools, while the remaining 56 tasks are held back for a potential competition leaderboard to limit the risk of models being trained on benchmark questions.
For engineering firms, the results suggest current agents can provide meaningful assistance on document-heavy work but are not yet suitable for unsupervised operation where a missed constraint carries real consequences. The article concludes that the constraint-pass rate, rather than the mean score, is the metric to track.
Claude Opus 5.5, developed by Anthropic, led the field with a mean score of 50.6%, averaged across three runs. OpenAI's GPT-6 Astra followed at 48.3%, and Google's own Gemini configurations scored in the low-to-mid 40s. The researchers emphasized the constraint-pass figure: the top agent satisfied every hard engineering constraint in only 41.3% of design tasks. The article notes that these constraints govern outcomes such as structural integrity and electrical safety, and that a mean score can reward partial credit even when a design violates a non-negotiable requirement.
The top performance carried a substantial compute cost. Claude Opus 5.5 consumed roughly 27 million tokens and needed 63 minutes per task, equating to approximately $17 in inference costs per task. The research framing states that meeting stringent constraints is where the difficulty lies and that even leading models have substantial room to improve.
The paper appeared on arXiv around October 21-23, 2026, under identifier arXiv:2610.30537. Google published a public GitHub repository containing 29 development tasks and supporting tools, while the remaining 56 tasks are held back for a potential competition leaderboard to limit the risk of models being trained on benchmark questions.
For engineering firms, the results suggest current agents can provide meaningful assistance on document-heavy work but are not yet suitable for unsupervised operation where a missed constraint carries real consequences. The article concludes that the constraint-pass rate, rather than the mean score, is the metric to track.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.