BabelArena tests AI agents on the same tasks in 23 languages

The QwenLM group released BabelArena, a test for AI agents. An AI agent is software that uses tools to do tasks. The test has 699 tasks in 23 languages, in all 16,077 task versions. The group used 5 frontier models in the test. Low-resource languages such as Lao, Khmer and Burmese had more errors and more token use.

How it works

BabelArena changes 4 other agent tests into 23 languages. A workflow with the name BabelFlow finds the text that is safe to translate. It translates prompts, tools, databases, simulated users and graders in the same procedure. Reviewers examine each version.

The numbers

• Mean translation quality is 4.28 out of 5 for high-resource languages and 3.91 for low-resource languages, on 4,600 samples.
• BabelFlow cut tasks with broken references from 20.0% to 2.5%.
• No model is first in all 4 test groups.

What is not known

The text does not name the 5 models. It gives no score for each model. The paper has the full error and cost analysis.

Insight: A count of languages is not a score

ElevenLabs gives more than 90 languages as the range of its new voice agent models. BabelArena shows that, for agents, low-resource languages give more tool errors and more token use. This reads as a distance between the number of languages and the results that a user must have. Tell the supplier to give results in your languages.