Robots.txt Generator — With AI Crawler Presets
Free robots.txt generator: pick a CMS preset (WordPress, WooCommerce, 1C-Bitrix, Laravel, Next.js, plus a note for Shopify), add your own Allow/Disallow rules, choose what to do with AI crawlers — block AI training (GPTBot, ClaudeBot, Google-Extended…) while keeping AI search, or block them all — and add your Sitemap. The file is checked for common mistakes and built right in your browser, no sign-up.
Configuración
Checked bots get “Disallow: /”. robots.txt is a request, not a lock: well-behaved bots follow it, anything else has to be stopped at the server or CDN (e.g. Cloudflare's AI bot blocking).
robots.txt
Upload the file to your site root (https://your-site/robots.txt) and test the live file with the robots.txt checker. The AI Visibility Checker shows which AI bots can reach you, and the llms.txt generator builds a short site summary for AI.
Datos clave
robots.txt is a text file in the site root where you ask crawlers to stay out of some sections. Pick a preset (WordPress, WooCommerce, Bitrix, Next.js and more), add your paths and Sitemap, optionally block AI bots, then copy the result. It is a request, not protection: well-behaved bots follow it, others do not, and a blocked page can still end up indexed.
- Precio
- Gratis
- Registro
- No necesario
- Procesamiento
- En tu navegador: no se envía nada a nuestro servidor
- Entrada
- A site preset, Disallow and Allow paths (one per line), the Sitemap URL, a Crawl-delay checkbox and a list of AI bots
- Límites
- No sign-up and no limits; it only generates and checks text, and you must upload the file to your site yourself
- Resultado
- A ready robots.txt text with a Copy or Download button and a list of warnings (Allow/Disallow conflict, blocking the whole site, Crawl-delay)
- Última actualización
Código abierto en GitHub
Ejecuta la misma comprobación en tu propio equipo: una herramienta de línea de comandos de Node.js, una biblioteca y una GitHub Action. Código abierto con licencia MIT.
Robots.txt generator with AI crawler presets
This robots.txt generator builds the rules file for search engines and AI crawlers: presets for WordPress, WooCommerce, 1C-Bitrix, Laravel and Next.js, a note for Shopify, your own Allow/Disallow paths, a Sitemap line and a separate block for AI bots. You can block AI training crawlers (GPTBot, ClaudeBot, Google-Extended and others) without touching the AI search bots that cite and link to your site. The result is checked for common mistakes as you type.
Cómo usar
- Pick your CMS preset or “Custom” and adjust the Allow and Disallow paths
- Decide what to do with AI crawlers: leave them, block training only, or block them all
- Add your sitemap URL and review the warnings under the file
- Download robots.txt, upload it to your site root and test the live file
FAQ
PromoPilot frente a alternativas: comparación
Comparamos solo lo que está confirmado en las páginas de cada servicio. Donde no pudimos confirmar un dato, la celda muestra "—".
| Parámetro | PromoPilot | Google Search Central: спецификация robots.txt |
|---|---|---|
| CMS presets | ✓ WordPress, WooCommerce, Bitrix, Laravel, Next.js | ✗ |
| AI bot list | ✓ 14 bots | ✗ |
| Rule conflict check | ✓ | Rule: the most specific wins; on conflict the least restrictive |
| Runs in the browser | ✓ | — |
Desplaza la tabla hacia los lados
Dónde somos más débiles: we do not check a published file or specific URLs, there is no preset for Joomla, Drupal and other CMSs, and the AI bot list needs periodic checking against the operators’ official pages.
Cuándo elegir PromoPilot: When you want a correct robots.txt from a CMS preset quickly and a check for typical mistakes. To check a published file and test URLs use the robots.txt checker or Search Console.
Comprobado el: . "—" significa que no está confirmado en la página oficial; las condiciones y los precios cambian, así que consulta los sitios de cada servicio.