GPT-6 Astra success decreases from 67% to 25% when a task changes

Researchers make SWE-Shift, a test for AI code agents when code, tools or requirements change during a task. It has 120 scenarios in 12 change families. Each scenario has one test with no change and one test with a related update. All 17 systems have less success after the change. The success of GPT-6 Astra decreases from 67.35% to 24.79%.

How it works

The two tests start from the same history and workspace. The update comes at one of nine points in the task. An agent must finish the task, keep correct work, and agree with the checks of the condition and evidence.

The numbers

• GPT-6 Astra: the paired success score is 20.34%.
• Changes to tools or sessions: no paired success on all systems.
• Expired evidence: no paired success on all systems.

Updates near the end of the task give less paired success for some systems and more for other systems.

What is not known

The authors give no cause for the low scores. The report gives no results for the other 16 systems.

Insight: Scores on tests that do not change are a maximum

Snowflake reports that almost a third of training tasks in three open data sets accept an incorrect patch. SWE-Shift shows that success on a task with no change can be true together with incorrect work after an update. The two results read as a maximum: scores on tests that do not change are more than the results when the work changes. Tell vendors to show results with changes in the middle of a task.