Benchmarks
Mathematicians find a mismatch in the OpenAI Navier-Stokes proofs
In one lemma, the Lean proof includes more values than the proof in natural language.
newscientist.comClaimed, not confirmed
News category
Benchmarks
7 stories, newest first.
Benchmarks
TestJack finds that 34.4% of trials shown as correct are incorrect
The researchers find that the resolution rate of the agents is 33.2% and not 50.6% when TestJack does the check.
arxiv.orgClaimed, not confirmed
Benchmarks
Reruns of the same tasks change 15 to 36 percent of the results
Tool names with incorrect information lower the reward of each agent by 0.20 to 0.44, the TRACE test shows.
arxiv.orgClaimed, not confirmed
Benchmarks
An evolved verifier had 0.21 more held-out agreement on MBPP+
Without its guards, the verifier always passes, but the skills that it trains are as good.
arxiv.orgClaimed, not confirmed
Benchmarks
HarmBench does not measure one safety attribute, researchers find
3 of the 4 datasets that can target harmful refusal are saturated and cannot show differences between models.
arxiv.orgClaimed, not confirmed
Benchmarks
Grok 4.7 is first in the new legal AI test with 9.4%
Artificial Analysis says a hallucination is material if it can mislead a reader, for example with an incorrect contract date.
Benchmarks
AI agents wrote GPU code with 1.57× the speed of `torch.compile`
The author tells that the correct test was more work than the fast code.
towardsdatascience.comClaimed, not confirmed