Gemini 4 Argon leads IBM Java migration test at 75.2%

IBM and Kaggle operate ScarfBench, an IBM test of if AI coding agents can move enterprise Java applications between frameworks (sets of prepared code). The applications must keep their operation. In 204 tasks, Gemini 4 Argon with Antigravity had 75.2% on the primary measure. Kaggle ran the test on its platform, pays for the compute and will update it for new models.

How it works

The test covers Jakarta EE, Quarkus and Spring migrations. It checks that the new application builds, starts and works as the test checks. Each result is a model with a harness, the tool that lets the model work in the code.

The numbers

• Gemini 4 Argon with Antigravity: 75.2% on the smoke score, which gives credit that is not full.
• Claude Opus 5.5 with Claude Code: 72.1% of tasks with all tests passed, more than each other pair.
• 535 of 745 runs with a failure broke at runtime.

What is not known

IBM made the test. Results are for pairs of model and harness, not for the model only. The first position held when 4 tasks that no pair completed were removed. No pair with all tests passed more than about 75% of the migrations.

Insight: Agent code ships fast, but faults show late

Researchers found that 43.1% to 56.2% of agent pull requests (code changes sent for merge) that change code and deployment setup together merge with no inspection. ScarfBench finds that most failures show only at runtime, after the code builds. This reads as a difference: inspection can check that code builds, but faults show when the application operates. Leaders who let agents migrate code must have tests of behavior.