Prompt-injection gates sent more than 91.8% of inputs to a reviewer

Researchers did a test of 4 defenses against prompt injection on 47,611 held-out examples. A prompt injection is text in an input that tells an AI model to do a task that its owner did not tell it to do. When the target for missed attacks was 1%, each system sent more than 91.8% of inputs to a reviewer. The authors write that a probability score is only a part of a good security decision.

What was in the test

The group did a test of 4 systems:

• Jev, a probabilistic decision model
• Llama Prompt Guard 2, the 22M classifier
• Llama Prompt Guard 2, the 86M classifier
• An LLM judge

The 47,611 examples had direct and indirect injections, hard benign inputs and obfuscation. They also had test data that is different from the training data.

The results

The LLM judge had the most AUROC of the 4 systems, a score for how correctly a system divides attacks from safe inputs. Its score was 0.897. The 22M classifier had a score of 0.714, which is less than all other systems.

Temperature scaling made the confidence of a model more correct. It did not give more AUROC. The probabilities of Jev were not always more correct than the corrected scores of the classifiers.

The cost

When the target for missed attacks was 1%, all systems had escalation of more than 91.8%. They also had high rates of safe inputs that the system marks as attacks.

The authors write that a team must do a test of detection, calibration, reviewer behavior and decision policy together.