Four AI firms back one token that stops AI crawlers

OpenAI, Google, Microsoft and Anthropic say that they will keep their AI crawlers away from a site that shows the token. A crawler is software that reads web pages to collect data. Microsoft gives publishers two selections, one for AI training and one for AI grounding, and no start date or test results are given.

How it works

A site shows the token, and the four companies say that a crawler must stay out when it finds the token. Training is the use of data to make a model. Grounding is the use of a verified source of information by a model, to decrease incorrect answers.

The proof

There are no numbers and no tests. The four companies only tell of their next steps. No external group checks that the crawlers obey.

What is not known

It is not known who made the token. It is not known how a site can see that a crawler obeys. Microsoft tells that a site stays in Bing search results with each selection.

Insight: Sites get a crawler control, but agents face other gates

The four companies will obey one token for crawlers. In the last weeks, Yelp, Delta and two banks stopped AI agents. Yelp lets in only agents from firms that buy its data license. This reads as follows: crawlers get one control, while agents face a gate that each site sets and can sell. Monitor if the token starts to include agents.