Guardrail safety scores change with the policy used for labels

Researchers wrote a paper that shows the safety score of a guardrail changes with the labels in its training data. A guardrail is a model that stops dangerous prompts. The team did training of 10 guardrails on the same 53,000 prompts. The prompts had labels from 2 definitions of what is dangerous. On the HarmBench copyright test, scores went from 0 to 100.

How it works

The same prompts got 2 sets of labels. One set shows if the prompt is for dangerous content. The other shows if a moderator can stop it. The team made 10 guardrails with the same architecture, at 2 scales. A change to 0.03% of the parameters moved a score across its full range.

The numbers

• The label definition caused a 23x range of scores between 2 tests.
• 2 over-refusal tests were 88% and 82% in the training data.
• Validation score follows the policy of the annotator (correlation 0.74), not detection skill (0.34).

What is not known

This is one paper, and no other group has done a check of it. It uses data from one industrial deployment. The authors write that the teacher of a student model is a cause that no group records in released checkpoints.

Insight: A safety score shows whose definition of dangerous was used

Previous tests of guardrails show that results change with the test, and that many stops come from check errors. This paper adds that the label policy and test data in the training set can set the score. A score reads as a result of a policy, and the party that writes the labels controls the result. Buyers who get the definition get control.