A check of 217 million web responses found 7,959 sites with AI headers

Researchers read the HTTP response headers of 217,097,223 pages from Common Crawl. This is 10% of its data of September 2026. They found 7,959 sites that sent a header written for an AI system. Most headers tell the agent to go to a different resource. A small number give instructions to the agent. The fetch tool controls which parts of a response go to the model.

What the data shows

Of the 7,959 sites, 7,271 sent pointers. A pointer is a header that tells the agent where to go next. Most used the header x-llms-txt.

Injection text

30 sites sent injection text. Most of it was a prank. Some text tells the agent to do a task. This works only if the agent can use a shell or a database.

The agent tests

In tests of October 2026, 3 Claude models did not obey the test headers by default. Then the instruction told them to obey the directions of the site. In 7 of 9 tests, they obeyed.

What to do

• Find which parts of a response your fetch tool sends to the model.
• Make each pointer destination a new fetch with no trust.
• Look at the result of each fetch in the agent. Do not look at header names.