A small scout model decreases routing cost by 9 to 33%

Researchers show how to decrease the cost of a router, the software that selects one AI model for each query. Routers use one cost that does not change for each model, but the output of reasoning models changes by about 10 times across queries. The authors use a small frozen model, the scout, to predict success and output length. On 8 of 9 datasets, this decreases the cost by 9 to 33% at the same accuracy, against a price at the median length.

How it works

The scout reads the query. A predictor, a small add-on model, reads the internal signals of the scout. For each model, it predicts if the result will be correct and how many tokens the model will write.

The numbers

• Full knowledge of cost gives a decrease of 29 to 61% on 8 of 9 datasets.
• The predictions get a maximum of 69% of that decrease.
• The predictor keeps most of its decrease with 20 labelled problems.
• The decrease stops at a scout of about 2B (2 billion) parameters.

The cost with the scout is lower than with ZeroRouter and a predictor of the MixLLM type on most datasets.

What is not known

On math and code, problems that are not easy have more output tokens, and predicted success is sufficient. On knowledge tasks it is not. The predictor works without labels on only 4 of 7 new benchmarks.

Insight: The router is becoming the control point for AI cost

In October 2026, vLLM Semantic Router released 5 open decision models. Other authors said router logs help an attack find 2.7 to 9.4 points more of the examples used to tune a model. A router seems to be a toll gate: it sees each query and sets the price, and attackers can use it. The scout gives it more data. Leaders: use the router as a system that holds cost and data.