Teams of 30 small AI models cut a single model's error by only 1.7%

Researchers tested the idea that many small language models give a better answer when their answers are averaged. They used 2.51 million answers from 15 open-weight models, from 3B to 72B parameters, on Fermi questions. These questions have rough estimates as answers. The models have the same error on each item, and a team of thirty gains a small amount.

How it works

The authors divide the error of each answer in two parts. One part is the mean error of the model on that item, the same for all its answers. The other part is a residual. The first part is a median 0.906 of the error variance, and more agents do not remove it.

The numbers

• Teams of thirty cut error by a median 1.7%, at most 5.4%.
• Three rounds of debate cut the variation in a team by 93%, but not the shared error.
• Five mixed models of equivalent precision gave 6.6% less error. No mix was better than the best single model.

What is not known

The results hold only for the models, tests, prompts and conditions used. The authors say that a higher sampling temperature changed team error by about 1% at most.

Insight: Agreement between agents is not proof

Agents in a team here have the same error on each item, and debate makes them agree without making them correct. ResolveLab, an open-source test tool, grades an agent from server records and not from the claim of the agent. This reads as a rule: the check that counts is external to the agent team, and its owner holds the control. Do not buy precision with more agents.