Llama 3.3 added 135 security flaws, and developers added 12

Researchers tested AI tools that repair software on more than 20,000 GitHub issues (reports of a problem), and updated the paper on 1 November 2026. The model Llama 3.3 added 135 flaws that persons found by hand, but developers added 12 in their repairs of the same issues. All 4 agent tools in the test also added flaws.

How it was tested

The team compared developer repairs, 2 models alone and 4 agent tools. An agent tool is control software that lets a model work in steps. The team also looked for the code and the issues that are connected to flaws.

The numbers

• Llama 3.3 added 135 flaws, and developers added 12.
• GPT-5.4 nano added 3 on the test set, and developers added 2.
• OpenHands added the most agent flaws: 44 in 11 issues.

What is not known

The results hold for this SWE-bench setup only. Flaws occur more with wide code changes and with issues that miss data. The authors say that repair systems must have a risk check of the code and of the issue text.

Insight: Agents add flaws and attackers can steer them

A different paper showed that changed variable names steered coding agents to write the flaw that an attacker wanted, in 22.5% to 56.0% of tests. Phoenix Security sells a check in the coding assistant, but gave no test results. This reads as a race to own the check in the agent, because agents add flaws and attackers can steer them. Tell the vendor to show proof on your code.