Test of 17 AI moderators finds no model strong on all five trust axes

Researchers released TrustMod-SM, a benchmark of about 29,000 instances in eight data sets that tests large language models (LLMs) as social media moderators. The team tested 16 open weight models and one closed frontier model on five dimensions: trustfulness, fairness, safety, robustness and context integrity. No single model was good on all five, and larger models did not remove all the problems.

How it works

The models have 0.5 billion to 284 billion parameters. The test checks more than accuracy. It checks bias, confidence in incorrect answers, and how a model handles tricky prompts.

The results

• Models frequently follow prompts to hide or exaggerate content.
• They give high confidence to incorrect answers.
• They rarely escalate items that are not clear.
• Below 8 billion parameters, they find it hard to tell hate speech from counter speech.

What is not known

The abstract gives no score for each model. It does not name the closed model. Larger models did not remove all the problems.

Insight: Accuracy alone hides the risk in AI moderation

The paper finds that models with higher accuracy frequently show more demographic disparity. Vendors usually sell moderators on accuracy. This reads as a trade: the number that wins the sale can hide the damage, and the buyer carries the risk. Buyers who request results for each group and for escalation rules can set the terms.