The top AI coding agent fully solves only 26.4% of tasks

Researchers made Co-Design Bench, a new test of AI coding agents. It uses requests from coding sessions of persons, tasks that were hard for agents before. The researchers tested 10 models. The top model, GPT-6 Astra, meets about 70% of the requirements of a task. Its result is fully correct in only 26.4% of tasks.

How it works

The tasks range from small changes to a full rebuild of a project. One AI judge tests the complete app in a browser. The other AI judge checks each correct verdict again and looks for incorrect results. Points go only to features that work.

The numbers

• Claude Opus 5 and Qwen3.8-Max follow GPT-6 Astra. Grok 4.6 is last.
• GLM-5.2 uses almost 6 times as many tokens (units of text) as GPT-6 Astra on small changes and has a lower score.
• Qwen3.8-27B got a score of 75% or 56% on large rebuilds, in two 2 different environments.

The authors tell all to report the model and its environment together.

What is not known

Each task has one set task text. The test has no follow-up tasks and no corrections during a task. The AI judges can make errors.

Insight: The test decides which agent looks best

Other tests give scores above 99% when models can check their code. This test gives 26.4% to the top model. A related test of AI infrastructure tasks gave 38.9% to its top model. Scores fall when a test copies work of persons. The environment can change which model is first, so the author of a test controls the sequence of suppliers. Leaders must test the pair on their tasks.