A new benchmark tests if guardrails follow authorization
Many blocked operations come from check errors and not from correct denials.
Claimed, not confirmed
Researchers made a benchmark for guardrails of language-model agents. A guardrail must stop a task that has no authorization and let the same task go with correct permission. An added reviewer removes some errors but also rejects correct authorization.
Sources
Posted
