The top AI coding agent solves 38.9% of AI infrastructure tasks

Researchers made InfraDev, a benchmark (a standard test) with 36 tasks that come from changes to tools that train and operate large AI models. Each task tells a coding agent to make a change across many files and many GPUs (AI processors), when older tests include only 1 GPU. The top of 6 frontier models solves 38.9% of the tasks.

How it works

The authors add probes, which are tests for the requirements that the initial tests do not check. The input text goes to the agent and names each interface that the probes use, but it does not give the solution.

The numbers

• The top of 6 models solves 38.9% of the tasks.
• The probes reject 32% of the changes that the official tests accept.
• With the initial change text as the input text, no model solves a task.
• A written agent skill increases the rate of 1 model by 10.5% on Megatron-LM tasks.

What is not known

The most frequent errors are in structure and state. The tests that the agents make do not frequently check memory, time or traffic. The text does not tell if the 10.5% increase is the same on other tools.

Insight: Who writes the checks sets the score

The probes reject 32% of the changes that the official tests accept. Arize tells teams to score each agent record with code checks, AI judges and human labels. A pass rate shows the skill of the grader as much as the skill of the agent. The person who writes the checks controls the score, thus find out who wrote them before you select a tool.