How to Block AI Bots in robots.txt (and Whether You Should)

To block AI bots in robots.txt, add one group that lists the bots and a Disallow: / rule. The most common setup blocks the crawlers that collect data for model training (GPTBot, ClaudeBot, CCBot and others, plus the Google-Extended and Applebot-Extended tokens) and leaves AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) alone: your content stays out of future training data, while links to your site in ChatGPT, Claude and Perplexity answers keep coming. Google Search is not affected. Keep in mind, though, that robots.txt is a request that well-behaved bots honour, not a technical barrier.

Below: the kinds of AI crawlers, ready-made blocks for three scenarios, the mistake that hands "allowed" bots access to your admin area, and how to verify the result.

Three groups of AI crawlers

"AI bot" covers very different robots. Decide on each group separately:

This split comes from the operators’ own documentation. OpenAI says the GPTBot and OAI-SearchBot settings are independent: you can allow search and disallow training. Anthropic describes its three bots the same way. According to Perplexity’s docs, PerplexityBot is not used to crawl content for training foundation models.

The third group comes with a caveat. OpenAI warns that robots.txt rules may not apply to ChatGPT-User because a person initiates the request. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic, by contrast, lets you control Claude-User through robots.txt.

Google-Extended and Applebot-Extended are tokens, not crawlers

These two names are often mistaken for crawlers, but neither of them fetches your pages.

• Google-Extended, per Google’s documentation, has no user agent of its own: regular Google crawlers do the fetching, and the robots.txt token only says whether the content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal.
• Applebot-Extended, per Apple’s documentation, does not crawl either. It decides whether data collected by the regular Applebot may be used to train Apple’s foundation models. Pages that disallow Applebot-Extended can still appear in Apple’s search features — Siri, Spotlight and Safari.

The practical consequence: you cannot block these two at the server by user agent, because no request carries that user agent. Blocking Googlebot or Applebot instead would hurt your search visibility.

Ready-made robots.txt blocks

Scenario 1. Block AI training, keep AI search

This is what the "Block AI training, allow AI search" preset in our robots.txt generator produces. Search crawlers are not mentioned, so they follow the general * rules:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Several consecutive User-agent lines form one group — shorter and harder to get wrong. Note that Amazon uses Amazonbot for its own services as well as for model training.

Scenario 2. Block all AI crawlers

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: Amazonbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /

The price is that links to your site disappear from AI search answers. For a business site that usually does not pay off; for a members-only knowledge base or paid content it may.

Scenario 3. Allow everything

Add nothing. AI crawlers that are not named in the file follow the User-agent: * group. A separate group with Allow: / is not just redundant but harmful — see the next section.

Scenario 4. Block AI from part of the site

For example, paid content and PDFs. Copy all your general rules into the AI group, or they stop applying to those bots:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /premium/
Disallow: /*.pdf$
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Why you should not add a separate Allow group for AI bots

The rule that breaks most hand-written files: a crawler obeys only one group — the most specific one for its name. That is how Google’s robots.txt spec and RFC 9309 describe it: if a bot has its own group, it ignores the * rules. Look at this file:

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/

User-agent: OAI-SearchBot
Allow: /

The author wanted to "explicitly allow" ChatGPT search, and instead gave OAI-SearchBot access to /wp-admin/ and /cart/: it found its own group, so the * restrictions no longer apply to it. That is why our generator never outputs allowed AI bots as a separate group. If a bot really needs its own group, repeat all the general restrictions inside it.

Within a group, a second rule applies: the longest matching path wins, and on a tie Google picks the less restrictive rule, i.e. Allow. More on syntax and common traps in checking robots.txt.

Should you block AI crawlers at all?

There is no universal answer, but the logic is clear:

• Blocking training makes sense when the content is your product: paid material, original writing, databases. It works going forward — Anthropic, for instance, says a block signals that the site’s future materials should be excluded from training. robots.txt does not recall data already collected.
• Blocking AI search rarely pays off for sites that live on traffic: ChatGPT search, Claude and Perplexity answers cite sources, and citations bring visits.
• Google-Extended is a separate call. It does not affect Google Search, but according to Google it also controls grounding in Gemini Apps, so blocking it may reduce your presence in Gemini answers.
• Server load. If AI crawlers strain a small server, that is a reason to limit them. Anthropic supports the non-standard Crawl-delay directive; Google does not. How server load relates to Googlebot crawling is covered in crawl budget.

robots.txt is a request, not a lock

Well-behaved bots follow the file; nothing technically stops others from ignoring it or faking a user agent. Cloudflare’s docs say it plainly: robots.txt compliance is voluntary. If you need to really stop bots:

• Block at the server. For example, by user agent in nginx (this only works for bots with their own user agent — not for Google-Extended or Applebot-Extended):

if ($http_user_agent ~* "(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent)") {
    return 403;
}

• Verify by IP. User agents are easy to fake. OpenAI, Anthropic, Perplexity and Apple publish the IP addresses or ranges of their crawlers, which lets you tell a real bot from an impostor.
• Use your CDN. Cloudflare offers a managed robots.txt for AI crawlers (available on all plans, per its docs) and a separate enforced block for AI bots (AI Crawl Control). Check the details of your plan in their documentation.

One caveat from Anthropic: if you block its IP addresses, the crawler may be unable to read your robots.txt, so your opt-out may not be reliably honoured. Set up robots.txt in any case, even if you also block at the server.

llms.txt is something else

A /llms.txt file neither blocks nor allows access. It is a proposed format for a short Markdown map of your site for language models: what the site is and which pages matter most. It cannot keep AI out, but it can help AI understand your site. Details in what is llms.txt; you can create one with our llms.txt generator.

How to verify the result

1. Open https://your-site/robots.txt in a browser and make sure the new version is live (CMSs, plugins and CDNs sometimes serve their own file).
2. Run it through the robots.txt checker: it lists all User-agent groups, Allow and Disallow rules and Sitemap lines, and warns about things like Disallow: / for every bot.
3. Check the site in the AI Visibility Checker. It parses robots.txt per RFC 9309 for 14 AI crawlers and tokens (who is allowed, who is blocked), checks for /llms.txt and the homepage meta robots, and sends a test request to the homepage with the GPTBot user agent. That test is only a hint: the real GPTBot comes from OpenAI’s IPs, and a CDN may treat it differently. The bot-access section is free; AI search answers to your questions count towards a daily limit (10 fresh runs a day without an account, three times that with one, unlimited on Pro).
4. After a few days, check your server logs: which user agents still arrive, and from which IPs. Changes are not instant — OpenAI, for example, says search systems can take about 24 hours to adjust.

Common mistakes

• A separate Allow: / group for "good" AI bots — it opens everything you closed for *.
• Blocking AI search together with training without noticing that links from ChatGPT and Perplexity answers have disappeared.
• Expecting Google-Extended to remove the site from Google Search. It does not affect Search; to remove a page from the index you need noindex — the difference is explained in noindex, nofollow and robots.txt.
• Trying to block Google-Extended at the server by user agent — there is no such user agent.
• Forgetting subdomains. robots.txt applies only to its own host: blog.example.com and shop.example.com need their own files.
• Typos in bot names and outdated lists. Names and purposes change, so re-check the operators’ documentation every few months.