Claude Opus 5.5 leads a new 100 task coding test with 67.3%
Claude Opus 5.5 got 67.3% and GPT-6 Astra got 63.7%, and the difference is not statistically important.
Researchers released ExpertCrowd-SWE, a test of 100 software tasks that engineers wrote for coding agents, in 87 open source projects and 11 languages. The team tested seven frontier models five times each, in 3,495 runs with scores. Claude Opus 5.5 led with 67.3%, and no model fully solved all tasks, because the team selected hard tasks.
How it works
Each task is a change request on a repository of a project that persons use. A sealed reference solution and a grader that gives credit for part of a solution go with it. Blocker rules mark what a maintainer refuses to merge.
The numbers
- Claude Fable 5.1: 46.5%. Gemini 3.8 Flash: 40.0%.
- Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6: 32% to 34%.
What is not known
On 80 tasks, no model got full credit in all runs. The number of steps for each run did not follow the score. The team selected the tasks because they are hard for current models.
Insight: The leader changes with the test
An earlier test, Co-Design Bench, put GPT-6 Astra first, and it fully solved only 26.4% of tasks. Here it is second, with a small difference. Each new test is built to be hard for the best models, and the rank depends on the test. Teams that own hard tasks of their own get the most control.
Sources
Posted
