What matters in AI.

Subscribe

Learn / AI agents

Definition · AI agents

Web crawler

A web crawler is software that automatically visits web pages, downloads them, and finds their links. Then it visits those links. Search engines use crawlers to make their index. AI companies use crawlers to collect training data for their models. They also use crawlers to get new pages for AI search and chat answers.

Last reviewed

Key points

  • A web crawler downloads a page, finds its links, and adds them to a list of pages to visit next.
  • AI companies operate different crawlers for model training, for search, and for user fetches. OpenAI has GPTBot, OAI-SearchBot and ChatGPT-User.
  • Site owners tell crawlers what to skip in a robots.txt file (RFC 9309). The standard says these rules "are not a form of access authorization".
  • A crawler can give an incorrect name. Make sure of a crawler from its published IP ranges or reverse DNS, not from its user agent string.
  • In Cloudflare network data for the 12 months to August 2025, 80% of AI crawler traffic was for model training.

How a web crawler works

A crawler starts from a list of known addresses. Google calls this list its “known pages”. Then the crawler does these steps again and again:

  1. Read the robots.txt file of the site, and skip the paths that it blocks.
  2. Download the page. Googlebot also operates the JavaScript of the page in a recent version of Chrome.
  3. Find the links in the page, and add the new links to the list.
  4. Select the next pages. Google says an algorithm decides “which sites to crawl, how often, and how many pages to fetch”.
A crawler reads the rules of a site, downloads a page, and adds the new links to its list of pages to visit.
A crawler reads the rules of a site, downloads a page, and adds the new links to its list of pages to visit.

Site owners control crawlers with the Robots Exclusion Protocol, published as RFC 9309 (September 2022). Martijn Koster first defined it in 1994. The rules are in /robots.txt at the top of the site. A crawler gives its name in its user agent string.

Why web crawlers are important for AI

Longpre and colleagues write that general purpose AI systems are built on “massive swathes of public web data”. For GPT-3, filtered Common Crawl data gave 410 billion tokens, 60% of the training data mix (Brown et al., 2020). Common Crawl has collected petabytes of web data since 2008.

Crawlers do three different tasks for AI companies:

Task OpenAI Anthropic Google
Collect training data GPTBot ClaudeBot Google-Extended (a robots.txt token, not a crawler)
Make an AI search index OAI-SearchBot Claude-SearchBot Googlebot
Get a page for one user ChatGPT-User Claude-User (not in this table)

Each name has its own robots.txt rule. A site can block training and continue to let search crawlers in. Sources: OpenAI, Anthropic and Google crawler documentation, read November 2026.

In practice

4 security risks of web crawlers

  1. An incorrect name. A client can send a user agent string such as GPTBot or CCBot. Common Crawl warns that other crawlers sometimes use the CCBot name. Make sure of a crawler from its published IP ranges or reverse DNS.
  2. Rules that a crawler does not obey. On 4 August 2025, Cloudflare reported that Perplexity used an undeclared crawler after a block. That crawler used a Chrome on macOS user agent and IPs that were not in the official Perplexity range. Cloudflare counted 3 to 6 million such requests each day across tens of thousands of domains.
  3. Poisoned pages. A crawler copies what a page shows. An attacker who controls a crawled page can use it for web-scale data poisoning of a training set. Hidden text on the page can also cause indirect prompt injection when a model reads that page.
  4. Too many requests. A crawler can send many requests to one site. Anthropic lets sites use the “non-standard Crawl-delay extension” to slow its bots.

Numbers on AI crawler traffic

  • 80% of AI crawler traffic was for training in the 12 months to August 2025. Search was 18% and user fetches 2% (Cloudflare network data, 29 August 2025).
  • 38,066 crawls by Anthropic for each visit it sent back to a site in July 2025. OpenAI: 1,091. Perplexity: 195. Google: 5.4 (Cloudflare, same report).
  • 45% of C4 had crawler restrictions in the Terms of Service of its sources by 2024. C4 is a training corpus (Longpre et al., audit of 14,000 domains, July 2024).
  • More than one million Cloudflare customers chose a one-click AI crawler block after September 2024. From 1 July 2025, Cloudflare blocks AI crawlers by default on new domains.

Trade-offs for site owners

Decision Result
Block training crawlers only Pages stay in AI search. Training use is refused, if the operator obeys.
Block all AI crawlers Fewer AI search referrals.
Block at the network by verified IP Stops crawlers that do not obey robots.txt. Needs upkeep of IP lists.
Let all crawlers in Most AI visibility. No control of training use.

Longpre and colleagues say that web protocols are “ineffective” and “not designed to cope with the widespread re-purposing of the internet for AI”.

Questions and answers

Does blocking GPTBot or ClaudeBot remove my site from ChatGPT or Claude answers?

Not fully. A robots.txt block on GPTBot or ClaudeBot tells the company not to use your pages for training. To keep out of search answers, also block OAI-SearchBot or Claude-SearchBot. OpenAI says that robots.txt rules "may not apply" to ChatGPT-User, because a user starts those fetches.

Can robots.txt stop a crawler?

No. RFC 9309 says robots.txt rules "are not a form of access authorization". A crawler obeys them only if its operator decides to. To stop a crawler, block its verified IP ranges at the server or network. Or put the content behind a login.

What is the difference between a web crawler and a web agent?

A web crawler collects many pages, usually for an index or a training set. A web agent opens pages to do one task for one user, such as a search or a purchase. OpenAI says its ChatGPT-User agent is "not used for crawling the web in an automatic fashion".

Sources

  1. RFC 9309: Robots Exclusion ProtocolIETF (Koster, Illyes, Zeller, Sassman), Sep 2022
  2. In-depth guide to how Google Search worksGoogle
  3. Overview of OpenAI CrawlersOpenAI
  4. Does Anthropic crawl data from the web, and how can site owners block the crawler?Anthropic
  5. Google's common crawlersGoogle
  6. Common Crawl overviewCommon Crawl
  7. CCBotCommon Crawl
  8. Language Models are Few-Shot LearnersarXiv (Brown et al.), 28 May 2020
  9. Consent in Crisis: The Rapid Decline of the AI Data CommonsarXiv (Longpre et al.), 20 Jul 2024
  10. The crawl-to-click gap: Cloudflare data on AI bots, training, and referralsCloudflare, 29 Aug 2025
  11. Perplexity is using stealth, undeclared crawlers to evade website no-crawl directivesCloudflare, 4 Aug 2025
  12. Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-LargeCloudflare, 1 Jul 2025

In the news