A detector of AI agent commits is right only 33% of the time

Researchers studied 7,837,946 commits, which are code changes in a repository, from 1,226 open-source repositories. They removed signatures, such as bot accounts, and tested if a detector can find the commits of 5 AI coding agents and of persons. At the agent share in the data, the top detector is right in only 33.2% of its flags.

How it works

Agents such as Claude Code and Cursor commit code to open-source repositories. Studies find these commits with signatures. A signature is optional, and a person can remove it.

The researchers masked the signatures and made detectors with the other data. Most of the signal came from message length and change size, not the code.

The numbers

• The top ranking score (ROC AUC) was 0.859.
• The top precision, the share of flags that are correct, was 0.332.
• On held-out Claude Code commits, TF-IDF reached a score of only 0.524.

What is not known

Detection gives a low score for agents that the detector did not see before. The Copilot score drops and then goes up again. The authors say signature counts are a minimum for agent work.

Insight: Counts of agent code may be too low

Jellyfish found that in September, AI agents opened or committed code to 27.7% of pull requests at the median company. In January it was about 2%. This paper says labels give only a minimum. Because the writer can remove an optional label, the control reads as a move to the platform that makes a record. Leaders must read a share of AI code that uses tags as a minimum.