The repository has more effect on agent pull requests

Researchers examined agent pull requests and human pull requests in 280 public repositories on GitHub. A pull request is a set of code changes that a person gives to a project to add. About 40% of the variation in the time to close an agent pull request is between repositories. The variance is 1.76 times as large as for human pull requests (95% confidence interval 1.41 to 2.18).

How it works

The researchers used multilevel models. These models divide the variation into parts: the pull request, the agent and the repository. They compared agent and human pull requests in the same repositories. The agent data is from the AIDev dataset.

The numbers

For a repository where waits are 1 standard deviation above average, the typical wait is about 2.8 times as long for agent pull requests. For human pull requests, it is 2.2 times as long. This is for waits of some hours. Pull requests that were rejected show the same pattern. The difference in rounds of inspection and in comments cannot be found.

What is not known

The role of the developer who started the agent gives only a small part of the difference. The number of agent policies was too small to test. The researchers write that oversight of coding agents must use the repository as a unit.

Insight: The area where an agent works changes its score

A paper shows that about 40% of the wait for agent pull requests is between repositories. In a test of one model, the type of remote changed the pushes from 65% to 54%. This reads as a sign that the area where an agent works changes its score, and a score with no area is weak. A merge rate can measure the repository, not the agent.