AI Bots in robots.txt: 7,875 Sites, 2026
Many sites now block AI robots, but often treat two different kinds of bot as one. Training bots take content for models. Search bots decide whether a site can be shown in answers from ChatGPT search, Perplexity or Claude. We checked the robots.txt of 7,875 popular sites in 10…
Many sites now block AI robots, but often treat two different kinds of bot as one. Training bots take content for models. Search bots decide whether a site can be shown in answers from ChatGPT search, Perplexity or Claude. We checked the robots.txt of 7,875 popular sites in 10 domain zones and found that sites block the first kind far more often than the second.
Data: crawl of November 10, 2026
Key numbers
- 19.7% of sites (1,551 of 7,875) block at least one AI training bot by name; 8.5% (669 of 7,875) block at least one AI search bot. Googlebot is blocked by 0.1% (9) and Bingbot by 0.3% (24). Crawl of November 10, 2026.
- 58% of the sites that block training bots (896 of 1,551) leave all 4 main AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) open. Of 1,109 sites that block GPTBot, 29% (321) also block OAI-SearchBot.
- By domain zone, at least one training bot is blocked by from 29.7% of sites in .it (532 sites) down to 10.9% in .br (403 sites). In .pl it is 24.9% for training bots but only 5.1% for search bots (587 sites).
- In a live check of 2,616 home pages, a request named GPTBot was refused by 12.8% (336) and a request named Googlebot, our control, by 6.8% (178). Of 247 sites that refused GPTBot but let Googlebot through, 56% (139) do not block GPTBot in robots.txt.
- llms.txt exists on 13.1% of sites (1,031 of 7,875): 18.6% of international sites and 5.9% in .fr. Whether AI systems use the file, we did not measure.
Two kinds of AI bots
The names look alike, but the purpose differs. Blocking a training bot means your content is not offered for model training. Blocking a search bot means that service may not find or cite your pages in its answers. The division below follows each company's own documentation.
| Bot | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | training |
| OAI-SearchBot | OpenAI | search and answers |
| ChatGPT-User | OpenAI | on a user's request |
| ClaudeBot | Anthropic | training |
| Claude-SearchBot | Anthropic | search and answers |
| PerplexityBot | Perplexity | search and answers |
| Google-Extended | Gemini models (training and grounding) | |
| CCBot | Common Crawl | training datasets |
One point causes the most confusion. By Google's documentation, Google-Extended does not affect Google Search or AI Overviews: it is a separate switch for use in Gemini models. Googlebot stays the one that matters for Search.
Who blocks whom
Among training bots the leaders are CCBot at 14.6% (1,149 of 7,875), GPTBot at 14.1% (1,109) and Bytespider at 14.0% (1,105), then ClaudeBot at 12.7% (1,002), meta-externalagent at 11.6% (913), Applebot-Extended at 10.8% (850) and Google-Extended at 10.7% (842). Search and answer bots are blocked less: PerplexityBot 7.0% (549), ChatGPT-User 5.8% (458), OAI-SearchBot 4.1% (326) and Claude-SearchBot 3.3% (260). Ordinary search engines are almost never blocked. Another 2.3% of sites (185 of 7,875) close everything with a wildcard rule, and they are not counted in the figures above, which count only bots named in the file.
Training is blocked, search is left open
Of the 1,551 sites that block at least one training bot, 896 (58%) keep all 4 main AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) open. That looks like a deliberate split: do not give content to models, but do not leave answer engines either. The picture is less tidy for GPTBot. Of 1,109 sites that block it, 321 (29%) also block OAI-SearchBot, so these sites may not appear in ChatGPT search. The owner may want that, or may simply have copied a list of bot names without separating the two kinds. The file cannot tell us which.
Zones differ
| Zone | Sites | Training bots | Search bots |
|---|---|---|---|
| .it | 532 | 29.7% | 16.2% |
| .de | 580 | 26.7% | 13.4% |
| .pl | 587 | 24.9% | 5.1% |
| .uk | 425 | 23.8% | 13.4% |
| .fr | 543 | 23.6% | 12.0% |
| .es | 473 | 23.0% | 12.7% |
| international (.com .org .net .io) | 3,216 | 17.2% | 7.5% |
| .pt | 522 | 15.7% | 4.6% |
| .ua | 594 | 12.6% | 2.2% |
| .br | 403 | 10.9% | 3.5% |
Blocking is most common in .it, .de, .pl, .uk, .fr and .es (23.0% to 29.7% for training bots) and least common in .ua and .br (12.6% and 10.9%). A zone is not a country: these are sites registered in a zone, not companies of a nation. The case of .pl stands out: 24.9% block training bots, but only 5.1% block search bots (587 sites). We do not know why .it and .de block more often. One possible reason is the position of European publishers on the use of their content by AI companies, but that is a hypothesis, and we did not test it. Correlation is not cause.
The server versus robots.txt
A rule in robots.txt is a request, and the server can say no on its own, through a CDN or a firewall. For this we ran a live check on 2,616 home pages that returned 200 to a browser. We then asked each page four more times, naming GPTBot, ClaudeBot, PerplexityBot and Googlebot.
| Bot name in the request | Refused |
|---|---|
| GPTBot | 12.8% (336) |
| ClaudeBot | 13.3% (349) |
| PerplexityBot | 9.7% (253) |
| Googlebot (control) | 6.8% (178) |
The control matters. All requests came from our own IP, not from OpenAI, Anthropic or Google addresses. A site may refuse because it sees a fake bot, not because the real one is blocked. That is why we sent a fake Googlebot too: its 6.8% shows the background level of protection against fakes. The gap above that level is what points at a rule against AI bots by name. 11.9% of sites (310 of 2,616) refused at least one AI bot and let the fake Googlebot through. Of 247 sites that refused GPTBot but let Googlebot through, 56% (139) do not block GPTBot in robots.txt. A refusal to our fake GPTBot is not proof that the real GPTBot is refused. What the real bots receive, we do not know.
llms.txt: present, but is it used
An llms.txt file exists on 13.1% of sites (1,031 of 7,875). Among international sites it is 18.6% (599 of 3,216), in .ua 12.1% (72 of 594), in .br 12.2% (49 of 403), in .pl 10.6% (62 of 587), in .de 7.6% (44 of 580) and in .fr 5.9% (32 of 543). The file existing does not mean AI systems use it, and we did not measure that. Two more signals in robots.txt: a Content-Signal line, which states permitted uses of content and is promoted by Cloudflare, is on 5.2% of sites (409 of 7,875), and the Cloudflare managed-rules marker on 2.9% (230 of 7,875).
What to do
The sample cannot say what is right for a given site, but these steps follow from it:
- Decide separately for training and for search. These are two decisions, not one.
- Write the rules by bot name. A minimal example that blocks training and leaves search open:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: /
The list is shortened; the full one depends on which bots you want to keep out. - Check that the CDN or firewall does not cut the search bots anyway, for example the Bots/WAF section of your CDN panel and logs by user agent. A rule in robots.txt does not help if the server refuses them.
- Do not block Googlebot to keep out AI: Google-Extended is a separate rule.
- Re-read robots.txt after changing CDN or host: managed rules and defaults may add or remove lines.
- Add llms.txt if you like, without expectations.
- Revisit the list once a quarter: new bots appear.
You can check your own robots.txt for free: robots.txt checker by PromoPilot. It shows which rules apply to which bots.
Methodology and limits
Sample: the Tranco list (downloaded November 10, 2026). International part: the first 5,000 domains in .com, .org, .net and .io. Country part: the first 700 domains in each of 9 zones (.uk .de .fr .it .es .pt .pl .ua .br). Of 11,300 domains, 7,875 answered: robots.txt returned 200 as text, or 404/410, which means everything is allowed. The 3,425 that did not answer are mostly technical domains (CDN, API, trackers). The sample is weighted toward popular sites and is not random. Collection ran on November 10, 2026 from one IP in Europe with the user agent PromoPilotResearch/1.0; if HTTPS did not answer we retried with www. Files were parsed by RFC 9309: a bot follows the group with its own name, or the wildcard group if there is none. A bot counts as blocked by name when its own group closes the path "/". Partial blocks (single sections) are not counted. The bot division follows the documentation of OpenAI, Anthropic, Perplexity and Google. The live check covered 3,300 home pages (1,500 international and 200 per zone), of which 2,616 returned 200 to a browser request. A refusal is a response of 401, 403, 406, 429, 451 or 503, or a dropped connection.
Limits: we read rules, not how they are applied; some companies say that agents opening a page at a user's request (ChatGPT-User, Perplexity-User) may not follow robots.txt. The server response depends on IP, time and bot protection, and this is one measurement. Site types (media, shops) were not labelled, so we do not compare them. Zone is not the audience country.
FAQ
What is the difference between GPTBot and OAI-SearchBot?
According to OpenAI's documentation, GPTBot collects content that may be used to train models, while OAI-SearchBot is used to find and show sites in ChatGPT search. They are separate robots.txt rules. In our sample 14.1% of sites (1,109 of 7,875) block GPTBot by name and 4.1% (326 of 7,875) block OAI-SearchBot.
If I block GPTBot, will my site disappear from ChatGPT?
Not by that rule alone. Search is handled by a different bot, OAI-SearchBot. Of 1,109 sites that block GPTBot, 29% (321) also block OAI-SearchBot, and those sites may not be shown in ChatGPT search. The other 71% do not block it. We did not measure what ChatGPT then shows, so this is a description of rules, not of results.
Does Google-Extended affect my rankings in Google?
By Google's documentation, no. Google-Extended only controls whether content is used in Gemini models and does not affect Google Search or AI Overviews. In our sample 10.7% of sites (842 of 7,875) block it, while Googlebot is blocked by 0.1% (9 of 7,875).
Does a site need an llms.txt file?
It is optional. 13.1% of sites in our sample have one (1,031 of 7,875). We did not measure whether AI systems read it, so we cannot say it changes anything. It is cheap to add, but expect nothing from it.
How can I check whether my server lets AI bots in?
Check two things: the rules in robots.txt, and the real server response to a request with the bot's name. In our live check of 2,616 home pages, 11.9% (310) refused at least one AI bot while letting a fake Googlebot through. Our requests came from our own IP, so the result is a hint, not proof about the real bot. Your server logs and CDN settings show what the real bots receive.
How many sites block AI bots in 2026?
In our sample of 7,875 popular sites, 19.7% (1,551) block at least one AI training bot by name and 8.5% (669) block at least one AI search bot. The sample is weighted toward popular sites and is not all sites on the internet. Crawl date: November 10, 2026.
How to cite
PromoPilot (2026). AI Bots in robots.txt: 7,875 Sites. Data November 10, 2026. CC BY 4.0: https://promopilot.link/uploads/blog/research/ai-bots-1211/ai-bots-robots-txt-2026-11-10.csv . Article: https://promopilot.link/en/blog/ai-bots-robots-txt-7875-sites-2026
Download the data
The aggregate table (CSV, license CC BY 4.0): https://promopilot.link/uploads/blog/research/ai-bots-1211/ai-bots-robots-txt-2026-11-10.csv. It contains aggregates by domain zone, not lists of domains.
