AI code auditors accept incorrect safe notes from a tool

Researchers test if AI models that check C and C++ code accept the notes of a code analysis tool. They do 16,572 audits on 916 pairs of functions, with eleven model configurations from four vendors. Correct notes increase accuracy by only 0.02. Incorrect notes that tell the model that the code has a check decrease bug detection by a maximum of 19 points.

How it works

The authors make a tool that reads one function and writes notes. The models use the notes as evidence. The authors change only the notes.

Some incorrect notes tell the model that the code has a check when the check is missing. Other incorrect notes tell the model that an added check is missing.

The numbers

• Incorrect assurances: bug detection decreases by a maximum of 19 points.
• Incorrect alarms: no important decrease. The models accept them less frequently.
• A check of each claim in a different call increases accuracy by 12 points.

Seven new models show the same pattern. Instructions in the prompt do not always decrease the error.

What is not known

For one other model, the gain from claim checks is less and not sure. The authors make the tool for C and C++ code only.

Insight: A note that tells all is safe is the weak point

The models in this study accept incorrect safe notes more than incorrect alarms. Stealthium reports that usual defense tools see a usual task when an attack changes model weights in the memory of the servers. The two results read as one pattern. An attack that looks safe or usual gets a smaller number of checks, and a quiet signal must have a check. Check each safe note before a model uses it.