BELLS-O tests 33 AI safety filters for speed and cost

Researchers released BELLS-O, a benchmark that has no link to a vendor and tests 33 systems from 17 developers on detection, speed and cost. The systems check text that goes into AI models, and find jailbreaks, which are input text that breaks model rules. The authors say that earlier benchmarks frequently favor vendors and do not give cost and speed.

How it works

The test compares specialized guardrails, such as Llama Guard 4, with general models, such as GPT-5.4. The general models operate as supervisors. The test has 11 categories of dangerous content and 13 jailbreak methods. It also has prompt injection, which is hidden commands in input text.

The numbers

• Moderation: the top systems of the two types find 93 to 96%, with no important difference.
• Jailbreaks: general models find more on three of four datasets, but closed frontier models cost 10 to 100 times more.
• Prompt injection: the two best dedicated detectors are in front.

What is not known

The authors made all the datasets. The cost depends more on the serving setup than on the type of system. The authors release the framework, a leaderboard and four datasets.

Insight: No single safety filter is best for every task

An earlier test found that a safety layer cut the success of a strong attack from 0.896 to 0.608. Attacks then continued to get through. BELLS-O shows that fast specialized filters do best in moderation, and costly general models do best in jailbreaks. This looks like a layered purchase: a fast filter for each task, and a costly model only for jailbreaks. Buyers get more when they test for each task.