Researchers decrease the cost of a check on AI answers
On an H100 processor, the check adds 16% to the latency at 2K tokens and 1.4% at 35K tokens.
Claimed, not confirmed
Researchers show a procedure to find AI answers that the given text does not support. A small frozen proxy model reads the text and the answer of a different model. The new procedure computes attention statistics without storage of the full attention matrix. The authors report that a 4B proxy had an F1 score above that of all 9 judge models on 2 of 3 benchmarks. These results are from the authors only.
Sources
Posted
