In LiveGeo, the results of AI agents fall when a map changes
GPT-5.6 Sol had 18.1% correct results on path tasks with no change, and 1.3% when conditions changed during the task.
On 12 November 2026, researchers from the University of Oxford, the National University of Singapore and two other universities showed GeoWorld and LiveGeo. GeoWorld is a simulation of six cities where roads and positions change with time. LiveGeo is a test with more than 1,200 tasks, that is made on it. The team used nine top models in the test, and all nine had low results when conditions changed.
How it works
GeoWorld uses data from six cities. The changes occur with time, and not only after a step of the agent. The test has three parts:
- Basic tasks, such as find a location or plan a path.
- Tasks with many steps, such as a plan with many stops.
- Tasks where conditions change during the work.
The numbers
- On long plans with many stops, the top result is 8.7%.
- Claude Opus 5 went from 21.5% with no change to 1.3% with changes.
- Gemini 3.7 Flash kept the most: 26.2% with no change, 13.4% with changes.
What is not known
The authors tell that agents keep a fact as correct for all time and frequently do not check it again. They also tell that models see that a road is blocked but do not make a correct new plan. No external team shows the same results.
Insight: Tests with no change do not show real results
CyberArena tests agents in services that change, and the defenders frequently did not patch the next hole. LiveGeo shows the same on maps. This reads as a move to tests where conditions change, and a high result with no change does not always hold. Make vendors give results with changes.
Sources
Posted
