AI crawlers: which to allow, which to block, and what each costs

The most commonly blocked line in this whole area is Google-Extended, and almost nobody blocks it on purpose. It reads like a Googlebot variant, so it gets swept up in a rule aimed at something else, and the site quietly leaves AI Overviews and Gemini while staying in Google Search.

That is the shape of nearly every mistake here. Not a decision anyone made, but a line copied from somewhere, matching a bot whose job the author did not know. This guide is the reference that makes those jobs legible.

What is an AI crawler?

An AI crawler is an automated fetcher operated by a company building AI products. Grouping them under one name is the source of most of the confusion, because they do three unrelated jobs.

The three are not degrees of the same thing. They are different products with different consequences, and a rule that treats them alike will be wrong about at least two of them.

• Answer crawlers fetch a page so an assistant can read and cite it, often within seconds of someone asking a question. Block one and no amount of on-page work will produce a citation from that engine.
• Search crawlers build ranked indexes. Two of them also feed AI answers: Bing’s index is what Copilot reads, and Googlebot’s is what AI Overviews draw on.
• Training crawlers collect text for future models. Blocking one has no effect on whether you are cited today. It is a licensing choice, and a block here is not a mistake.

Answer crawlers: the ones that decide citations

These are the agents where a disallow line has an immediate, measurable cost. If you read nothing else, read this section.

• OAI-SearchBot (OpenAI) builds the index ChatGPT search draws on. Blocked, ChatGPT cannot read your pages. It may still name your brand from memory, but it has nothing to link to.
• ChatGPT-User (OpenAI) fetches a single page when a user’s prompt sends ChatGPT to open a link. Blocked, a user who explicitly asks ChatGPT to read your page is told it cannot.
• PerplexityBot (Perplexity) indexes pages so Perplexity can cite them. Blocked, you stop appearing in its sources, which is the only place Perplexity links from.
• Perplexity-User (Perplexity) fetches a page a user has asked to open. Blocked, they get an error.
• Google-Extended (Google) governs whether Gemini and Google’s AI answers may use your content. It fetches nothing itself. Blocked, your pages stay in Google’s index and drop out of AI Overviews.
• Claude-SearchBot (Anthropic) indexes pages so Claude’s web search can find and cite them.
• Claude-User (Anthropic) fetches a page when a Claude user asks it to read a link.

Notice the pattern across three of the four operators: an indexing bot and a user-initiated fetcher, under separate names. Blocking only the second produces the worst kind of bug report, where the site appears in answers but every user who tries to open it gets an error.

Search crawlers: two of them now feed AI answers

• Googlebot is the crawler behind Google Search and behind AI Overviews. If anything in this guide is a genuine emergency, it is this one.
• Bingbot builds the Bing index, which is also what Microsoft Copilot reads. You lose both together, which is a larger loss than Bing’s search share suggests.
• Applebot feeds Siri, Spotlight and Safari suggestions.

Training crawlers: blocking these is a licensing decision

Nothing in this section affects whether an assistant can cite you today. Allow or disallow them on the merits of the licensing question, which is a real question and not one this guide has a position on.

• GPTBot (OpenAI) collects text for future models. Frequently blocked in the belief that it controls ChatGPT citations. It does not.
• ClaudeBot (Anthropic) collects training text. Claude-SearchBot and Claude-User handle citation.
• Applebot-Extended (Apple) is a switch rather than a crawler: it governs whether content Applebot already fetched may train Apple’s models. Siri and Spotlight follow Applebot and are unaffected.
• Google-CloudVertexBot (Google) fetches for customers building their own Vertex AI agents, and only for sites they have asked it to read.
• CCBot (Common Crawl) builds the open web archive that most public training sets are cut from. It is the widest-reaching entry here, which makes it the one where a deliberate position is worth having.
• Bytespider (ByteDance) collects training data. Commonly blocked for crawl volume rather than for policy.

The robots.txt mechanics that decide the outcome

Most accidental blocks are not misunderstandings about bots. They are misunderstandings about how a robots.txt file is evaluated.

A crawler obeys exactly one group

A crawler picks the single most specific User-agent group matching its name and follows only that one. It does not merge groups, and it does not fall back to User-agent: * once a more specific group exists.

This is the rule that surprises people. If you add a permissive group for one bot, the restrictions in your wildcard group stop applying to it entirely, including the ones you meant to keep.

Matching is on substrings, not patterns

Agent tokens are matched case-insensitively, and a token matches when it appears in the crawler’s user-agent string. So User-agent: Google catches Googlebot and Google-Extended together, which is exactly how the accidental block described at the top of this guide happens.

Wildcards are not supported in agent names. GPT* matches nothing.

An empty Disallow allows everything

Disallow: with nothing after it permits the whole site. Disallow: / forbids all of it. One character apart, and both look deliberate in a diff.

The four failures worth checking for

1. A blanket disallow aimed at scrapers. A User-agent: * group with Disallow: /, added during a staging deploy or a scraping incident, and never reverted. This blocks every answer crawler at once.
2. Google-Extended caught by a Google rule. Described above. The site stays in search, so nothing looks broken, and the AI answers quietly stop.
3. Blocking GPTBot and expecting it to stop citations. The reverse of a mistake: people block it deliberately, then conclude AI search does not work for them. The citation path was never affected.
4. An indexing bot allowed and its user-fetcher blocked. Appears in answers, fails when a user clicks through to have the assistant read the page.

How to verify it

Fetch your own /robots.txt and read it against the rules above, remembering that the group a bot obeys is the most specific one matching its name rather than the first one in the file.

Doing that by hand for sixteen agents is tedious and easy to get wrong, which is why the free AI crawler checker exists: it resolves the group each agent would obey and reports whether the site is open, partially restricted or blocked for each one, with the matched User-agent line shown so you can see which rule decided it.

For the argument about which of these deserve your attention first, the shorter piece on which AI crawlers matter covers it in less detail and less time.

What this does not tell you

Crawler access is a precondition and not an achievement. Allowing every agent here makes you reachable. It does not make you worth citing, and the work that does is a different discipline covered in the guide to generative engine optimization.

Robots.txt is also a request rather than an enforcement mechanism. The operators listed here document their agents and honour the file. Bots that ignore it are not solved by editing it, and blocking a well-behaved crawler while a badly-behaved one continues is a common and unhappy outcome.

Finally, this list is a snapshot. Operators rename agents, split them, and add new ones without much notice. Anything that matters this much deserves rechecking on a schedule rather than once.