Top AI agent scores 48.3% on a Google test of physical engineering

Google announced the Physical-Engineering Last Exam, made with Mercor. It is a benchmark, a standard test for AI agents on mechanical, electrical, civil and structural engineering work. Google tested 14 frontier models on the private test set, and released 29 tasks to the public. All had scores below 50%, and the best had a mean score of 48.3%.

How it works

The test has 85 long tasks. Each task is a project folder with an average of 61 files, such as drawings, spreadsheets and emails. The agent must find building codes, calculate forces and make work that a licensed engineer can examine.

The numbers

Agents get points only if the work goes through all hard safety checks first. The top setup did this in 41.3% of its work. Models changed scores by a maximum of 50 points, but control software changed them by a median of 3 points.

Not yet known

The results are from tests made by Google. Prize instructions are not given at this time. Google says no earlier benchmark tests this work from start to end.

Insight: Google hosts the private test for engineering agents

Google released 29 of the 85 tasks and keeps the others private. It operates submitted agents on servers of Google, with 13 models that Google sponsors free. This reads as a bid to own the yardstick for engineering agents, and the first test that vendors use can set the rules. Tell vendors to show results on private tests.