A new plugin decreases false positives in patch checks by 25%

Trail of Bits has 2 new plugins: patching-bugs and regression-hunt. Engineers use them to give agents the work to repair bugs. The agents also look for regressions, which break code that was correct. In first tests, regression-hunt decreased false positives by 25%, compared with a basic Codex agent. They made the plugins with OpenAI in the Patch the Planet project.

How it works

The patching-bugs plugin reproduces a known bug, makes a repair and tests it. The regression-hunt plugin follows a change to the positions that use it. It tests each possible regression on a live copy of the system. A new agent does the same test again. If a step cannot make a live test, it gives no result.

The numbers

Its engineers examined all patches that maintainers merged. They found 21 regressions: 16 in function, 1 in speed and 4 in software assembly or test. None broke a security protection. The 25% number has a range of 14% to 38%.

What is not known

This proof step has a limit. Some known regressions stayed only possible, because the plugin did not collect sufficient indications to make sure of them. Trail of Bits did the tests, and the results are first results.

Insight: Set checks replace belief in the agent

Trail of Bits found 21 regressions in more than 130 accepted patches. Its step is set formats and a new agent that does each test again. 2 AWS engineers tell that hooks (control points in code) stop a defective agent. This reads as a change: the agent is not the proof, and set checks that are external to it are. The group that writes the checks sets the quality. Leaders must look for test proof.