A broken part in an AI gateway left its system score almost the same

Researchers wrote a paper on BARBICAN, a gateway between a model and its tools that blocks prompt injection (hidden orders in text that trick a model). One part of it, the arbitration stage, got text that a tokenizer (software that cuts text into pieces) had broken, for a full recorded test. After repair, the AUC score of that part rose from 0.392 to 0.855, but system recall fell from 80.67% to 80.34%.

How it works

The gateway blocks an input when it is not sure. Earlier parts send the arbitration stage inputs that are 96.82% malicious. A bad verdict blocks an input. A good verdict lets it go only through one narrow path.

The numbers

A stub that always says “malicious” scores 80.77% recall. This is higher than the broken stage and the repaired stage. A constant verdict in place of the stage changes only 12 to 140 of 25,000 decisions.

What is not known

The gateway leads on its own benchmark and falls behind when the data changes. The authors say the same is true for their earlier gateway. They say a stage score is hard to read if you do not know the share of attacks in its input.

Insight: One score can hide a dead layer

Stealthium gave no detection rate for its sensor that finds hidden edits to model weights. The BARBICAN paper shows a broken stage that left the system score almost the same. Vendors of AI defense seem to compete on one headline number, and that number cannot show if each layer works. Buyers can get tests that remove and replace each layer.