What matters in AI.

Subscribe

Open model Beveridge beats five large AI models at set decisions

It got 93.0% on the test of the authors, against 91.1% for the best of five general models.

Researchers released Beveridge, a family of open weight models (anyone can download the model). They make narrow decisions about a document, for example send a ticket to the correct team or check a limit. They do not write text. A model reads a document and a maximum of 15 questions. It gives an answer with a calibrated risk number for each choice in one pass. The weights and code use the Apache 2.0 licence.

How it works

The small version has 398M parameters (a measure of how large the model is). It adds a small decision network to ModernBERT-large. A larger version adapts Gemma 4 26B-A4B. A web version picks the next step of a browser agent.

The numbers

  • The small model got 93.0%. The best of five general models, 31B to 675B in size, got 91.1%.
  • Its confidence is calibrated. It can hand over 76% of decisions automatically at 95% precision.
  • No general model that the authors tested can hand over one.
  • It answers in 4.9 ms on one NVIDIA H200 GPU.

What is not known

The authors did all the tests, on 19 held out test sets, and the benchmark is their own. They say the paper shows where the models are weak. The abstract does not list these.

Insight: Small open decision models are now a crowded race

Microsoft released Decision-1 (10 October 2026) and Sarvam released Prism (3 November 2026). The two give a number for each choice and write no text. A paper (9 October 2026) found that a hosted decision model gave unsatisfactory likelihoods when errors cost different amounts. Each maker tests alone on its own benchmark. This reads as a race to be the standard fast gate for business software. Buyers must make their own test.

Sources

  1. Beveridge: Open Weight System One Decision Models with Calibrated Probabilitiesarxiv.org
AI MATTER · NEWS · AI MATTER · NEWS ·17 NOV2026

Posted

Tags

Learn the terms in this story

Guides for this story

More in Models

All Models news