A team of 5 small models solved 49 of 50 hard code runs
The team got a mean score of 9.8 of 10, but the test had only 10 problems.
A team of 5 small Qwen3.8-27B models solved 49 of 50 runs on 10 hard code problems. The test rebuilt the method of a 2026 paper by Lee. The team got a mean score of 9.8 in 10 for 5 passes. The set-up of the authors got 9.6. The model Fable 5 with no team got 8.0.
How it works
The team has a planner, idea makers, a solver and a manager. Each role starts new, with no chat history. It uses files on a disk to give information to the other roles.
A script does the sample tests. Only the script, not a model, gives the result.
The numbers
The total cost of the 50 runs was $35.79. A median run had a cost of $0.39.
What is not known
The test had 10 problems, not the 100 of the authors. The problems come from contests in 2025. The model can know them from its training data.
Fable 5 was not operated in this set-up. Its score comes from the files of the authors.
The loop has gaps. A manager can make an incorrect counterexample, and a correct result can be overwritten by the next step. The test authors report that these gaps are fixable, but the changes are not in the numbers.
Insight: The set-up around a model now drives the result
This team of small models got a score above that of Fable 5 with no team. A test bed compares set-ups for code agents, and Life-Harness lifts agents with no change to model weights. It looks like the value moves from the model to the set-up around it. For a company that selects AI, a good set-up can give more than a change of model.
Sources
Posted
