🔍 SEO / GEO

AI Crawlers

AI bots · AI web crawlers · LLM crawlers · AI user agents

Definition

AI crawlers are robots run by AI companies that crawl websites to train models, build AI search indexes or fetch a page on a user's request; they are controlled in robots.txt by bot name.

What AI crawlers are

AI crawlers are robots run by the companies that build language models and AI assistants. Technically they do the same as search bots during crawling — they download pages. The difference is what happens to the content next: it may go into a training dataset, into an AI search index or straight into an answer for the person who asked a question. That is why a single allow-or-block-AI rule is not enough; the decision is made for each type separately.

Three types of AI bots

Purpose	OpenAI	Anthropic	Perplexity	Google
Model training	GPTBot	ClaudeBot	none declared	Google-Extended (robots.txt token)
AI search index	OAI-SearchBot	Claude-SearchBot	PerplexityBot	—
User request	ChatGPT-User	Claude-User	Perplexity-User	Google-Agent

The names follow the companies’ documentation as of September 2026. Google-Extended differs from the rest: it is a token, not a separate bot. Crawling is done by Google’s regular crawlers, and the token decides whether the content may be used to train Gemini and for grounding. User-triggered fetchers are better classed as AI agent traffic: each visit is driven by a specific person’s request.

How much they take and how much they send back

Cloudflare compares how often a company’s crawler requests pages with how many referrals its product sends back. For July 2025 the ratios were:

Company	Crawler requests per referral
Anthropic	38,065
OpenAI	1,091
Perplexity	194
Google	5.4

Over the 12 months to July 2025, 80% of AI bot crawling was for training, 18% for search and 2% for user-requested actions.

A sample robots.txt for a store

An option for a store that wants links in AI answers (GEO) but does not want to hand its content over for model training: block training, allow AI search.

# Model training - blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# AI search - allowed, except service sections
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/
Disallow: /checkout/
Allow: /

# YandexGPT and Search with Alice answers - a separate decision
User-agent: YandexAdditional
Allow: /

Important: blocking Google-Extended also opts you out of grounding in Gemini, not just training. Make that decision deliberately.

Checklist

Find AI bots in your server logs and check their IPs against the companies’ published lists: the User-Agent name is easy to fake, and not every bot carries Web Bot Auth signatures yet.
Decide by type: training, search and user requests are different business questions.
Remember that user-triggered fetchers may not follow robots.txt: if you need to restrict them, do it at the CDN or WAF level.
After editing robots.txt, check its syntax and give bots time: OpenAI cites about 24 hours.

Frequently asked questions

If I block GPTBot, will my site disappear from ChatGPT?

No. According to OpenAI's documentation, the settings for its bots are independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot, which collects data for model training. Sites that block OAI-SearchBot, however, are not shown in ChatGPT search answers, although they can still appear as navigational links. OpenAI says robots.txt changes take about 24 hours to be reflected.

Does Google-Extended affect rankings in Google?

According to Google's documentation, no: the Google-Extended token does not affect inclusion in Google Search and is not used as a ranking signal. It controls whether content is used to train future generations of Gemini models and for grounding — supplying material from the index to the model at prompt time. It has no separate User-Agent: crawling is done by Google's regular crawlers, and the token is only read in robots.txt.

How do I limit use of my site in Yandex AI answers?

Yandex provides the YandexAdditional token for this: according to Yandex Webmaster help, it is taken into account when processing robots.txt to limit how page content is shown in YandexGPT quick answers and in answers from Search with Alice. General robots.txt rules do not apply to it, so it needs its own User-agent: YandexAdditional group. Yandex's main indexing robot is YandexBot, and its rules are set separately.

Do AI bots respect robots.txt?

According to their documentation, the training and search bots of OpenAI, Anthropic and Perplexity follow robots.txt, and Anthropic's bots also honour Crawl-delay. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Google describes Google-Agent the same way. Anthropic states that Claude-User does respect robots.txt.