What matters in AI.

Subscribe

In a test of 12 AI models, 75% of 1,200 language opinions were attacks

Python got 299 attacks, and models frequently select a language to attack.

A test gave 12 new AI models from 10 labs in 3 countries 2 questions about programming languages. Each model got each question 50 times, in 1,200 answers in all. The test used USD 7.68. Of the 1,200 opinions, 904 were attacks on a language and 291 were defenses. Each model went back to some of the same targets.

How it was done

Each model got the questions in a new chat with no system text. One question was about a random language. The other was about languages in general.

The numbers

  • Python got 299 attacks in 312 opinions.
  • PHP got 28 defenses in 28 opinions.
  • No opinion was about C#.

What is not known

The source does not tell if the results are the same for other questions or for code work. A model can also give 2 sides. GPT-6.1 Sol gave 20 defenses of Go errors and 21 attacks on them.

Insight: Many models do not give many views

9 of 10 labs gave the same opinion on Rust. Claude Opus 5.5 and GLM 5.3 wrote one sentence about Java word for word. But GPT-6.1 Sol gave the 2 sides on Go. This reads as one model with the same views each time, and many models are not a panel of experts. Use a model to find arguments, not to select.

Sources

  1. AI models hate Python and love PHP: we asked 12 of them 1,200 timesblog.jetbrains.com
AI MATTER · NEWS · AI MATTER · NEWS ·15 NOV2026

Posted

Tags

More in Research

All Research news