Robots.txt Generator — With AI Crawler Presets
Free robots.txt generator: pick a CMS preset (WordPress, WooCommerce, 1C-Bitrix, Laravel, Next.js, plus a note for Shopify), add your own Allow/Disallow rules, choose what to do with AI crawlers — block AI training (GPTBot, ClaudeBot, Google-Extended…) while keeping AI search, or block them all — and add your Sitemap. The file is checked for common mistakes and built right in your browser, no sign-up.
Configurações
Checked bots get “Disallow: /”. robots.txt is a request, not a lock: well-behaved bots follow it, anything else has to be stopped at the server or CDN (e.g. Cloudflare's AI bot blocking).
robots.txt
Upload the file to your site root (https://your-site/robots.txt) and test the live file with the robots.txt checker. The AI Visibility Checker shows which AI bots can reach you, and the llms.txt generator builds a short site summary for AI.
Fatos principais
robots.txt is a text file in the site root where you ask crawlers to stay out of some sections. Pick a preset (WordPress, WooCommerce, Bitrix, Next.js and more), add your paths and Sitemap, optionally block AI bots, then copy the result. It is a request, not protection: well-behaved bots follow it, others do not, and a blocked page can still end up indexed.
- Preço
- Grátis
- Cadastro
- Não é necessário
- Processamento
- No seu navegador — nada é enviado ao nosso servidor
- Entrada
- A site preset, Disallow and Allow paths (one per line), the Sitemap URL, a Crawl-delay checkbox and a list of AI bots
- Limites
- No sign-up and no limits; it only generates and checks text, and you must upload the file to your site yourself
- Resultado
- A ready robots.txt text with a Copy or Download button and a list of warnings (Allow/Disallow conflict, blocking the whole site, Crawl-delay)
- Última atualização
Código aberto no GitHub
Execute a mesma verificação na sua própria máquina: uma ferramenta de linha de comando em Node.js, uma biblioteca e uma GitHub Action. Código aberto sob a licença MIT.
Robots.txt generator with AI crawler presets
This robots.txt generator builds the rules file for search engines and AI crawlers: presets for WordPress, WooCommerce, 1C-Bitrix, Laravel and Next.js, a note for Shopify, your own Allow/Disallow paths, a Sitemap line and a separate block for AI bots. You can block AI training crawlers (GPTBot, ClaudeBot, Google-Extended and others) without touching the AI search bots that cite and link to your site. The result is checked for common mistakes as you type.
Como usar
- Pick your CMS preset or “Custom” and adjust the Allow and Disallow paths
- Decide what to do with AI crawlers: leave them, block training only, or block them all
- Add your sitemap URL and review the warnings under the file
- Download robots.txt, upload it to your site root and test the live file
FAQ
PromoPilot vs. alternativas: comparação
Comparamos apenas o que está confirmado nas páginas de cada serviço. Onde não conseguimos confirmar um dado, a célula mostra "—".
| Parâmetro | PromoPilot | Google Search Central: спецификация robots.txt |
|---|---|---|
| CMS presets | ✓ WordPress, WooCommerce, Bitrix, Laravel, Next.js | ✗ |
| AI bot list | ✓ 14 bots | ✗ |
| Rule conflict check | ✓ | Rule: the most specific wins; on conflict the least restrictive |
| Runs in the browser | ✓ | — |
Role a tabela para os lados
Onde somos mais fracos: we do not check a published file or specific URLs, there is no preset for Joomla, Drupal and other CMSs, and the AI bot list needs periodic checking against the operators’ official pages.
Quando escolher o PromoPilot: When you want a correct robots.txt from a CMS preset quickly and a check for typical mistakes. To check a published file and test URLs use the robots.txt checker or Search Console.
Verificado em: . "—" significa que não foi confirmado na página oficial; condições e preços mudam, então confira os sites dos próprios serviços.