The best AI agent setup plans 78% of tasks offline and 40% online

Researchers tested if AI agents can plan. A formal check decides if each step is applicable and if the goal is reached. They used five domains, four models, six prompting strategies and three agent designs. The best setup solves 78% of the solvable tasks offline, but only 40% online. Online, agents mostly work without a look at the condition of the task. The authors say that a claim of planning needs proof from checked steps.

How it works

Each task is in plain language. A hidden PDDL model, a formal description of a planning problem, decides if a step is applicable and if a plan exists. Two of the five domains are new. The authors release all five with the paper.

The numbers

• Offline, the best setup solves 78% of the solvable tasks. Online, it solves 40%.
• Agents find unsolvable tasks in less than 10% of episodes.
• Heavy prompting strategies use 5 to 22 times as many tokens as prompting without more steps.

What is not known

Results changed more across models than across prompting strategies or agent designs. The source does not name the four models. It does not tell how the results hold on tasks in other domains.

Insight: The model and external checks give results, not heavy prompts

This paper finds that the model is more important than the prompting strategy, and that heavy prompting costs much more. Authors of an earlier test report that a gate gives 95% correct tasks, not 65%. The money seems better spent on the selection of a model and on external checks than on longer prompts. Test your own agent with a checker that is not a judge model.