A Snyk agent passes 90.2% of dependency upgrades in a new benchmark

On 15 October 2026, Snyk released VulnBump, a benchmark that tests AI agents on 307 upgrades of dependencies that have a vulnerability. An upgrade passes only if the vulnerabilities are removed and the project installs, builds, and passes its tests. Snyk says its Evo Remediation Agent passed 90.2% of the upgrades. Claude Code passed 59.6% with no Snyk data on the same model.

The results

Snyk ran 3 agents on the Claude Sonnet 5.5 model. It also ran one baseline with no AI. Each rate is the part of the upgrades that passed:

• Evo Remediation Agent: 90.2%
• Claude Code with the Snyk upgrade and scan: 77.5%
• Claude Code with no Snyk data: 59.6%
• Package manager bump, with no AI: 46.9%

Where agents stop

All 3 agents made a project that builds about 93% of the time. The hard part was to remove the vulnerability. Some upgrades did not pass. In many of these, Claude Code with no Snyk data said that it removed the vulnerabilities, but a new scan found them.

How the test works

Snyk chose projects of the same type as those in its production data. A verifier copies the changes of the agent to a clean container. An upgrade passes only if all 4 checks pass:

• The dependencies install with the install command of the project.
• A new Snyk scan shows that the vulnerabilities are removed.
• The project builds.
• Each test that passed before passes again.

The limits

Snyk says it tested one model and ran each upgrade one time. It used its scan as the judge. Snyk says this can count only against Claude Code with no Snyk data. Snyk did not count new vulnerabilities that an upgrade added.