How Many Sites Block AI Bots in robots.txt (5,071, 2026)
On November 15–16, 2026 we read the robots.txt of 4,772 websites: 15.1% of them (719) tell at least one AI crawler to stay out of the whole site with a Disallow: / rule. The most blocked bot is ByteDance's Bytespider (12.0%), then Common Crawl's CCBot (11.9%) and OpenAI's GPTBot (11.5%); the…
On November 15–16, 2026 we read the robots.txt of 4,772 websites: 15.1% of them (719) tell at least one AI crawler to stay out of the whole site with a Disallow: / rule. The most blocked bot is ByteDance's Bytespider (12.0%), then Common Crawl's CCBot (11.9%) and OpenAI's GPTBot (11.5%); the bots that fetch pages for live answers and AI search are blocked far less often (OAI-SearchBot 3.6%). Separately, 22.5% of 5,071 sites serve a real llms.txt file, and sites that block AI bots have one less often (15.6% against 25.1%). Below: the ranked list, what each bot is and what blocking it does and does not do, the llms.txt picture, a decision table for your own site, and the methodology. Caveat: the sample is non-random (sites from our search-results cache), so the numbers describe this sample, not the whole web.
Data: PromoPilot crawl of 6,000 sites, November 15–16, 2026 (llms.txt check November 16); 661 sites behind a WAF, 178 unreachable and 90 with other errors excluded, 5,071 in the calculation, robots.txt found on 4,772
Key numbers
- 15.1% of sites (719 of 4,772 with a robots.txt) disallow at least one of the 10 AI bots we track. Of these, 121 (2.5% of all sites) also have a blanket
Disallow: /for every crawler, so the share that targets AI bots specifically is 598 sites, 12.5%. - Ranking by bot: Bytespider 12.0%, CCBot 11.9%, GPTBot 11.5%, ClaudeBot 10.8%, Google-Extended 9.5%, anthropic-ai 8.8%, Applebot-Extended 8.2%, PerplexityBot 5.7%, ChatGPT-User 5.3%, OAI-SearchBot 3.6%.
- Blocking is mostly all-or-nothing: of the 719 blocking sites, 379 (52.7%) block 7 or more of the 10 bots, 95 (13.2%) block all 10, and 100 (13.9%) block exactly one.
- Training versus live retrieval: 15.0% of sites block at least one training bot, 6.6% block at least one search or retrieval bot, and 8.4% block training bots only. Of the 549 sites that block GPTBot, 382 (69.6%) leave OAI-SearchBot open.
- Content-Signal (the robots.txt directive for search, ai-input and ai-train preferences) is present on 5.4% of sites (258); 90 of them use it without blocking any bot.
- llms.txt: a real file on 22.5% of 5,071 sites (1,139), llms-full.txt on 5.1% (259); 8.6% (435) answer 200 with an HTML stub or a redirect, which we do not count; 63.5% return 404.
- Sites that block AI bots serve llms.txt less often: 15.6% (112 of 719) against 25.1% (1,019 of 4,053 that do not block), among sites with a robots.txt.
- By CMS (generator tag): 17.6% of sites without a generator tag block at least one AI bot, WordPress 9.3% (462 sites), Drupal 19.1% (47), other CMS and builders 5.3% (587).
Ranked: how many sites block each AI bot
For each site we parsed robots.txt and recorded which of 10 AI user agents are disallowed from the whole site. The base is the 4,772 sites where a robots.txt was found (94.1% of the 5,071 we could read). The ranking is below.

| # | Bot | Sites blocking | Share of 4,772 |
|---|---|---|---|
| 1 | Bytespider | 571 | 12.0% |
| 2 | CCBot | 569 | 11.9% |
| 3 | GPTBot | 549 | 11.5% |
| 4 | ClaudeBot | 515 | 10.8% |
| 5 | Google-Extended | 452 | 9.5% |
| 6 | anthropic-ai | 421 | 8.8% |
| 7 | Applebot-Extended | 391 | 8.2% |
| 8 | PerplexityBot | 271 | 5.7% |
| 9 | ChatGPT-User | 254 | 5.3% |
| 10 | OAI-SearchBot | 173 | 3.6% |
Two patterns stand out. First, the first seven are all bots collected for model training or training-related permissions; the three lowest are the ones tied to search and answers. Site owners block "the bot that trains on my content" much more readily than "the bot that might send me a visitor". Second, the legacy anthropic-ai token (8.8%) is still blocked almost as often as ClaudeBot (10.8%): 384 sites block both and 552 (11.6% of all) block at least one of the two, which suggests many robots.txt files are copied from published block lists rather than written bot by bot.
Blocking is all-or-nothing for most sites
The 719 blocking sites do not choose bot by bot. Half of them (379) block seven or more bots, 95 block all ten, and only 100 block a single one.

One point on the figure itself. Our parser reports a bot as blocked when its rule disallows the whole site. 121 of the 719 sites also carry a blanket Disallow: / for all crawlers, which closes the site to search engines too (2.5% of all sites with a robots.txt; some of these are sites under construction). If you want the share that is specific to AI bots, subtract them: 598 sites, 12.5%.
Anatomy: what each bot is, and what blocking it does and does not do
"Blocking AI" is not one decision. The ten user agents belong to three different jobs: collecting pages to train models, building a search index that answers questions with citations, and fetching a page right now because a person asked an assistant to. Blocking each has a different effect. The descriptions follow the vendors' public documentation as we read it in November 2026; crawler documentation changes, so check the linked pages before you rely on a detail.
| Bot | Run by | Job | What blocking it does | What it does not do |
|---|---|---|---|---|
| GPTBot | OpenAI | Training crawler | Asks OpenAI not to use your pages for training future models | Does not remove you from ChatGPT search; that uses OAI-SearchBot |
| OAI-SearchBot | OpenAI | Search index for ChatGPT search results | Asks OpenAI not to include your pages in its search features | Not a training crawler; blocking it will not stop training |
| ChatGPT-User | OpenAI | Fetches a page when a user or a custom GPT asks for it | Asks OpenAI not to open your pages on user requests | OpenAI states that robots.txt rules may not apply to these user-initiated visits |
| ClaudeBot | Anthropic | Training crawler | Asks Anthropic not to collect your pages for training | Does not cover Claude-User (user requests) or Claude-SearchBot (search), which Anthropic lists separately |
| anthropic-ai | Anthropic | Older training token | Older token. Anthropic's current crawler docs list ClaudeBot, Claude-User and Claude-SearchBot; keeping anthropic-ai in the list does no harm | Many sites list it out of habit |
| Google-Extended | A robots.txt token, not a separate crawler; controls use of content for Gemini training and grounding | Opts your content out of those Gemini uses | Does not change Google Search crawling or indexing, and Google documents that it does not govern Search features such as AI Overviews | |
| Applebot-Extended | Apple | A token; controls whether content crawled by Applebot is used to train Apple's models | Opts your content out of that training use | Applebot itself keeps crawling for Siri and Spotlight search |
| PerplexityBot | Perplexity | Indexes pages for Perplexity's search | Asks Perplexity not to surface your pages in its results | Perplexity says it is not used to train foundation models; user-triggered visits use a different agent |
| CCBot | Common Crawl | Builds an open web archive that many AI developers use for training | Keeps new pages out of future Common Crawl datasets | Does not remove what earlier crawls already stored |
| Bytespider | ByteDance | Crawler for ByteDance products and models | Asks ByteDance not to crawl the site | Some site operators report that it ignores robots.txt; if that matters, block at the server or CDN as well |
Four things robots.txt cannot do
- It is a request, not a lock. A well-behaved crawler reads it; a badly behaved one does not. To enforce a block, use your CDN, firewall or web server rules (the 661 sites that answered our crawler with a WAF block page use this stronger tool).
- It is not retroactive. It affects future crawling. Content that a crawler already collected, or that sits in an older Common Crawl archive, stays where it is.
- Training and answers are separate switches. Blocking the training bots (GPTBot, ClaudeBot, Google-Extended) is a different choice from blocking the bots that decide whether your page can be cited in an AI answer (OAI-SearchBot, PerplexityBot). In our data 8.4% of sites block training bots only, 6.6% block both groups, and only 4 sites block retrieval bots alone.
- It does not hide a page from Google Search. Google-Extended is separate from Googlebot. Blocking the first leaves ordinary search crawling untouched, while a blanket
Disallow: /or a block on Googlebot closes the site to search.
Where sites split: training versus retrieval
Take OpenAI. Of the 549 sites that block GPTBot, 249 (45.4%) also block ChatGPT-User and 167 (30.4%) also block OAI-SearchBot; 382 (69.6%) block the training crawler but leave the search crawler open. That is a coherent policy: opt out of training, stay eligible to appear in ChatGPT search. Google's pair behaves similarly: 121 sites block GPTBot without Google-Extended, and 24 block Google-Extended without GPTBot.
The newer Content-Signal directive expresses the same split in a single line, with separate yes or no values for search, ai-input and ai-train. It appears on 5.4% of sites (258). Of those, 168 (65.1%) also block at least one AI bot by name, and 90 use the signal alone. We only record that the directive exists; whether a given crawler honors it is up to the crawler.
llms.txt: how many sites have a real file
llms.txt is a proposal from 2024 for a Markdown file at /llms.txt that lists the most useful pages of a site for language models, with short descriptions; llms-full.txt is the longer variant with the content inlined. It is not an official standard and, as of this writing, we are not aware of a major AI vendor that has confirmed it uses these files for retrieval or ranking, so treat it as low-cost documentation for machines, not a visibility lever. We checked it anyway, because it is the other half of the "what do I tell AI" question.
We count a "real file" only when the address answers 200, is not an HTML page and does not redirect to another page. That excludes 435 sites (8.6%) that answer 200 with an HTML page (a soft 404 or a stub) or a redirect.

| Measure | Sites | Share |
|---|---|---|
| Real llms.txt file | 1,139 | 22.5% of 5,071 |
| Real llms-full.txt file (all of them also have llms.txt) | 259 | 5.1% of 5,071 |
| 200 with an HTML stub or a redirect (not counted) | 435 | 8.6% of 5,071 |
| 404 | 3,218 | 63.5% of 5,071 |
| Real llms.txt among sites that block AI bots | 112 of 719 | 15.6% |
| Real llms.txt among sites that do not (robots.txt found) | 1,019 of 4,053 | 25.1% |
What the real files look like: the median is 5,922 bytes and the 90th percentile 56,975 bytes; 77.9% start with a heading (H1) as the format suggests; 89.0% contain at least one Markdown link (the median file has 24); 10.3% have none and are effectively plain text. 11.6% are served as text/markdown, the rest as text/plain.
The relationship with blocking is a correlation only: sites that block AI bots are less likely to publish a file written for AI systems (15.6% against 25.1%), which fits a simple reading that these sites are choosing not to engage with AI, while the others are choosing to. We cannot say that one causes the other; some files come from plugins and builders that create them automatically (the share is 35.3% among sites on non-WordPress CMS and builders, 25.5% on WordPress, 20.6% with no generator tag). Our own sample is also skewed to sites that appear in search results.
By CMS and by language
The CMS comes from the generator meta tag; 3,598 of the 4,772 sites do not publish one, and that is the largest and the most blocking group. Groups under 30 sites are not shown. Note: hostnames in our data are hashed, so a breakdown by domain zone (TLD) is not possible; we show the page language declared in the HTML as the closest geographic signal.

| CMS (generator tag) | Sites with robots.txt | Block any AI bot | Block GPTBot | llms.txt file (of all sites in group) |
|---|---|---|---|---|
| No generator tag | 3,598 | 17.6% (632) | 13.4% | 20.6% (793 of 3,856) |
| Other CMS or builder | 587 | 5.3% (31) | 4.6% | 35.3% (218 of 618) |
| WordPress | 462 | 9.3% (43) | 7.1% | 25.5% (120 of 471) |
| Drupal | 47 | 19.1% (9) | 8.5% | 10.6% (5 of 47) |
| MediaWiki | 42 | 4.8% (2) | 2.4% | 2.4% (1 of 42) |
| Joomla | 36 | 5.6% (2) | 2.8% | 5.4% (2 of 37) |
Read the CMS split with care. Sites with no generator tag are often custom builds, stores and larger commercial sites, and those are the owners most likely to have a policy; sites on ready-made platforms mostly keep the platform's default robots.txt. The difference is about who runs the site as much as about which software it uses. Drupal and MediaWiki groups are small (47 and 42 sites).
| HTML language | Sites with robots.txt | Block any AI bot | Block GPTBot |
|---|---|---|---|
| en | 1,655 | 12.6% (208) | 8.6% |
| ru | 823 | 7.2% (59) | 4.5% |
| uk | 516 | 10.1% (52) | 5.6% |
| no language declared | 505 | 34.7% (175) | 32.7% |
| sv | 336 | 13.7% (46) | 11.3% |
| de | 311 | 19.6% (61) | 17.4% |
| pl | 102 | 16.7% (17) | 9.8% |
| he | 89 | 11.2% (10) | 5.6% |
| es | 85 | 22.4% (19) | 20.0% |
| fr | 72 | 13.9% (10) | 11.1% |
| nl | 35 | 14.3% (5) | 2.9% |
German- and Spanish-language sites in this sample block more often than English-, Russian- and Ukrainian-language ones. The group with no declared language stands out at 34.7%, more than twice the sample average; we have not analysed why. Treat the language table as a description of this sample, not as a national statistic: the sample comes from the search queries we track.
How to decide for your site
There is no universally right setting. The decision depends on what you want from AI systems: to be found and cited, to keep content out of training, or both at once. Use the table as a starting point and then test the result.
| Your situation | Reasonable robots.txt stance | Watch out for |
|---|---|---|
| You want to appear in AI answers and search (service business, local business, B2B, shop) | Allow OAI-SearchBot, PerplexityBot and ChatGPT-User; allow or leave the training bots unlisted | Blocking the search bots removes you from the pool that could cite you; do not copy a "block everything" list by accident |
| You are fine being found but do not want your text used for training | Disallow GPTBot, ClaudeBot (and anthropic-ai if you like), CCBot, Google-Extended, Applebot-Extended; keep the search and user bots allowed | Not retroactive; some crawlers may ignore it; this is the policy 8.4% of sites in our data follow |
| Paid or members-only content, original research, a media archive | Disallow all training bots, decide on the retrieval bots case by case, and enforce with CDN or firewall rules, not only robots.txt | Robots.txt does not stop a crawler that ignores it; check your server logs |
| You want nothing from AI systems at all | Disallow all ten tracked bots (13.2% of the blocking sites do) and add server-level rules | Do not use a blanket Disallow: / for *: it also closes the site to search engines |
| You have no clear policy yet | Leave things as they are, add a Sitemap line, record which bots hit your server, revisit in a quarter | An unexamined copy-paste list is the most common source of accidental blocks |
A short checklist:
- Open your own
/robots.txtand look for the ten names above. Our free Robots.txt Checker parses the file and shows which rules apply to which bot. - Write the policy in one sentence (for example "allow search and answer bots, refuse training") and build the file from it. The Robots.txt Generator creates the groups for you.
- Make sure you did not block Googlebot by accident: AI rules should name the AI user agents, not
*. - If you publish an llms.txt, treat it as documentation: a title, a short summary and a list of key pages with one-line descriptions, served as plain text or Markdown with status 200. We have no tool for it, and a stub page that returns 200 is worse than a 404 because crawlers cannot tell it from a file.
- Review the policy when a vendor changes its documentation; user agent names change (anthropic-ai is a case in point).
Methodology and caveats
Data comes from our own TRI crawl of 6,000 sites on November 15–16, 2026: the corpus of sites that appear in Google results for the queries we track, plus users' sites and a control sample. Hostnames in the export are replaced by hashes and we publish aggregates only. We excluded 661 sites (11.0%) that answered the crawler with a bot-protection (WAF) page (420 of them Cloudflare), 178 unreachable sites (3.0%) and 90 with other server or HTTP errors (1.5%). The 655 WAF-blocked sites we retried after seven days were all blocked again. The calculation covers 5,071 sites; robots.txt was found on 4,772 (94.1%) and the percentages in the robots sections use those 4,772. The llms.txt shares use all 5,071.
A site "blocks" a bot when the parsed robots.txt disallows the bot's user agent from the whole site (Disallow: /). The ten user agents are the ones in the ranked table. Partial blocks (a folder) are not counted as blocks. Content-Signal and a blanket Disallow: / for all crawlers are counted separately. The llms.txt "real file" test is: status 200, content that is not HTML, no redirect to a different page.
Limits:
- The sample is not random. It is built from sites that appear in search results for our queries, so it leans towards SEO-active sites. Do not transfer the percentages to the whole web.
- Sites behind a WAF are excluded, and those are probably more protected and larger sites; their robots.txt policy may differ.
- Robots.txt shows the owner's intent, not whether a bot obeys it.
- We read rules as of one snapshot; sites change their files, and AI vendors add and rename crawlers. We track ten user agents and do not cover others (for example the newer separate agents some vendors document for user requests).
- The CMS comes from the generator meta tag, which most sites do not publish; the language comes from the HTML
langattribute. Domain-zone (TLD) breakdown is not available because hostnames are hashed. - Correlations (blocking and llms.txt, CMS and blocking) are not causes.
Two notes on the numbers. Our working draft of 16.11 described 379 sites as blocking "all 7 main bots"; the final count is 379 sites that block seven or more of the ten tracked bots, 95 of which block all ten. The draft also compared 15.6% with 23.6% for llms.txt, where the second figure used all sites without AI blocks including those with no robots.txt; here the base is the same (sites with a robots.txt), which gives 25.1%.
FAQ
Should I block AI bots in robots.txt?
It depends on what you want. If you want to be cited in AI answers, keep the search bots (OAI-SearchBot, PerplexityBot) allowed. If you object to your text being used for training, disallow the training bots (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended). In our sample of 4,772 sites with a robots.txt, 15.1% block at least one AI bot, and 8.4% block training bots only.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search results use OAI-SearchBot, and user requests use ChatGPT-User. Of the 549 sites that block GPTBot in our data, 69.6% leave OAI-SearchBot open.
Does blocking Google-Extended affect my Google rankings?
Google documents Google-Extended as a separate token that controls use of content for Gemini; it is not part of Googlebot and does not govern Search crawling or indexing. We do not claim any effect on rankings either way.
Do AI bots obey robots.txt?
The major vendors say their crawlers do, with the exception that OpenAI says user-initiated ChatGPT-User visits may not follow it. Robots.txt is a voluntary standard, and some site owners report crawlers that ignore it. To enforce a block, use your CDN or server rules.
Do I need an llms.txt file?
No vendor requires it, and we are not aware of a confirmed use of it by a major AI system. 22.5% of the 5,071 sites we checked publish one. It is cheap to add and harmless if kept accurate, but do not expect it to change visibility.
How can I check which AI bots my site blocks?
Open /robots.txt and look for the user agents named in this article, or paste the file into our Robots.txt Checker. To create a new file, use the Robots.txt Generator.
Sources
- OpenAI, Overview of OpenAI crawlers: platform.openai.com/docs/bots
- Anthropic, crawler and robots.txt help article: support.anthropic.com
- Google Search Central, Google's common crawlers (Google-Extended): developers.google.com
- Apple, About Applebot: support.apple.com/en-us/119829
- Perplexity, Perplexity crawlers: docs.perplexity.ai/guides/bots
- Common Crawl, CCBot: commoncrawl.org/ccbot
- IETF, RFC 9309, Robots Exclusion Protocol: rfc-editor.org/rfc/rfc9309
- llms.txt proposal: llmstxt.org
- Content Signals Policy: contentsignals.org
How to cite
PromoPilot (2026). How many sites block AI bots in robots.txt: 5,071 sites, November 2026. Data November 15–16, 2026. CC BY 4.0: https://promopilot.link/uploads/blog/research/ai-bots-robots-1911/ai-bots-robots-5071-sites-2026-11.csv . Article: https://promopilot.link/en/blog/how-many-sites-block-ai-bots-in-robots-txt-5071-sites-2026
Download the data
The aggregate tables (CSV, license CC BY 4.0): https://promopilot.link/uploads/blog/research/ai-bots-robots-1911/ai-bots-robots-5071-sites-2026-11.csv, plus the tables https://promopilot.link/uploads/blog/research/ai-bots-robots-1911/ai-bots-blocked-count-2026-11.csv, https://promopilot.link/uploads/blog/research/ai-bots-robots-1911/ai-bots-by-bot-2026-11.csv, https://promopilot.link/uploads/blog/research/ai-bots-robots-1911/ai-bots-by-cms-2026-11.csv, https://promopilot.link/uploads/blog/research/ai-bots-robots-1911/ai-bots-by-html-lang-2026-11.csv, https://promopilot.link/uploads/blog/research/ai-bots-robots-1911/llms-txt-by-cms-2026-11.csv. It contains aggregates only, no projects, domains or client data.