All 8 AI routers get a score less than GPT-6.1 Sol in a test
GPT-6.1 Sol has 61.2% at $15.4 for 1,000 questions, and no router has more than 56.2%.
A test uses 8 routers on 523 database tasks. A router is software that selects the AI model for each query. The report compares the score and the price of each router with one model. For each router, one model has the same score or more, at the same price or less. Switchyard from NVIDIA has a score 5.0 points less than GPT-6.1 Sol.
How it works
For each question, an AI agent selects one of 11 databases and writes SQL code. SQL is a language for database queries. The agent can send a maximum of 10 queries to a model. The router selects the model for each query. Some routers add a fee or a judge model to the bill.
The numbers
GPT-6.1 Sol has a score of 61.2% at $15.4 for 1,000 questions. Not Diamond has 50.9% at $50.5. Sakana Fugu Max has 27.3%, less than all other routers.
What is not known
The routers do not use the same pool of models, and the gaps show the router and its models. Sakana does not name its model. The report also gives a random mix test, but it does not end.
Insight: A change of model can add to the bill of a router
The OpenRouter manual shows that a change to a new model adds to the bill. The new model must make the cache again. The cache is the work that the model did before. A different test shows that agents send the full history each turn, and this work is done again. It reads as if a router gives a selection, but an agent keeps its cache with one model.
Sources
Posted