Which AI crawlers matter, and which ones you can safely block

A marketing director asks why the company stopped showing up in Gemini’s answers. Nothing has changed. Rankings are steady, traffic is flat, Search Console is clean, the content calendar shipped on time. Eight months earlier somebody tidied up robots.txt, added a block of AI user-agents they had read about, and included one line that was not like the others.

That line is Google-Extended, and it is the most expensive thing on this page.

Three jobs, one file

The reason this mistake is so easy to make is that robots.txt gives no indication that its entries do different things. They all look like bots. They are not.

Answer-engine crawlers fetch a page so an assistant can read and quote it in an answer it is composing now. Block one and you are not visible in that engine, full stop — it cannot read the page it would be citing. No amount of on-page work routes around it.

Search crawlers build the classic indexes. Two of them also feed AI surfaces: Bing’s index is what Microsoft Copilot reads, and Googlebot’s is what AI Overviews draw on.

Training crawlers collect text for future models. Blocking one is a licensing decision. It has no effect whatsoever on whether an assistant names you this afternoon, because the model answering this afternoon was trained a while ago and is looking things up live.

Those are three separate decisions that people make as one, in a single sitting, from a single list found on the internet. Two of them are cheap. One of them is not.

The agents, by operator

OpenAI

• OAI-SearchBot — builds the index ChatGPT search draws on. This is the one that decides whether ChatGPT can cite you.
• ChatGPT-User — fetches a single page when a user’s prompt sends ChatGPT to open a link. Blocking it means a person who explicitly asks ChatGPT to read your page is told it cannot.
• GPTBot — training data. Blocking it costs no citations.

The useful consequence: no training use, still citable is a coherent and fully supported position. Disallow GPTBot, allow OAI-SearchBot and ChatGPT-User. Most sites that blocked “OpenAI” blocked all three.

Google

• Googlebot — search, and by extension AI Overviews. If anything here is an emergency, it is this one.
• Google-Extended — not a crawler at all. It fetches nothing; Googlebot does the fetching. It is a switch controlling whether your content may be used by Gemini and Google’s AI answers.
• Google-CloudVertexBot — fetches pages for customers building their own Vertex AI agents. Not something you are measured on.

Google-Extended is worth dwelling on because of how it fails. Disallow it and your pages stay in Google’s index, keep their rankings, and keep their traffic. The only thing that changes is that Google’s AI surfaces stop using them. There is no warning, no Search Console message, and no report anywhere that says so. The signal is an absence, in a channel most companies were not measuring in the first place.

Perplexity

• PerplexityBot — indexes pages so they can appear as sources. Perplexity’s sources list is the only place it links from, so this is the whole of the visibility question there.
• Perplexity-User — fetches a page a user has asked it to open.

Anthropic

• Claude-SearchBot — indexes pages for Claude’s web search.
• Claude-User — fetches a page when a Claude user asks it to read a link.
• ClaudeBot — training. Same split as OpenAI’s, and the same opportunity to take a position on training without giving up citations.

Microsoft and Apple

• Bingbot — Bing’s index, which is also what Copilot reads. Blocking it loses you both at once, which is a bigger loss than Bing’s search share suggests.
• Applebot — Siri, Spotlight and Safari suggestions.
• Applebot-Extended — a switch, like Google-Extended: it governs whether content Applebot already fetched may be used to train Apple’s models. Blocking it leaves Siri and Spotlight untouched.

The dataset crawlers

CCBot builds Common Crawl, the open web archive most public training sets are cut from — the widest-reaching entry on this page, and so the one where a licensing position is worth taking on purpose rather than by default. Bytespider collects training data for ByteDance and is frequently blocked for crawl volume rather than for policy. Neither affects whether you are cited today.

Writing the rules

Two things about the syntax cause most of the damage. The first is that robots.txt is most-specific-wins, not additive: an agent with its own group ignores User-agent: * entirely rather than merging with it. If you write a group for PerplexityBot containing one Disallow: /admin, that group is now the whole of what Perplexity obeys, and your careful catch-all rules no longer apply to it.

The second is that a blank Disallow: means nothing is disallowed. It is how you open a group, not how you close one.

So the “no training, still citable” position reads:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow:

Everything not named is allowed — including OAI-SearchBot, Claude-SearchBot, PerplexityBot, Google-Extended and the search crawlers. That is the point.

What robots.txt cannot do

It is a request, not a lock. The major operators publish their user-agent names and honour the file; nothing in the protocol compels anyone to. If a page genuinely must not be read, robots.txt is the wrong instrument — it is a public list of what you would rather people did not fetch, and it is the first file a determined scraper opens.

It also is not the only way to be excluded. A noindex meta tag, an X-Robots-Tag header or a login wall will each keep a page out, and none of them appears in robots.txt. A file that looks perfect is necessary, not sufficient.

Do this now, then forget about it

This is a five-minute audit that stays correct for months, which makes it unusually good value. Read your file agent by agent and ask the only question that matters for each line: does this bot decide whether an assistant can quote me, or does it decide whether a future model is trained on me? Those are different questions, and one of them has an answer you can afford.

The AI crawler checker does exactly this. 16 agents, grouped by what blocking them actually costs, and for every block it names the line in your file that caused it — because a rule aimed at an agent by name and a Disallow: / inherited from the catch-all are the same verdict and two entirely different fixes.