Robots.txt Checker & Tester

How to test robots.txt online and read the report

To test a robots.txt file, enter your site address: the checker takes the host from it and requests https://host/robots.txt, following redirects. Subdomains are separate hosts with their own files, so check shop.example.com separately from example.com. If the file can't be fetched, you see why: 404 ("file not found"), another HTTP error code, or a network error.

When the file loads, the robots.txt analyzer shows a score from 0 to 100, the line count, the number of Allow and Disallow rules, the Sitemap list, a rules table for every User-agent and the raw file. The score drops as follows:

Two warnings carry no penalty: Crawl-delay above 10 seconds and a file larger than 512 KB (Google reads only the first 500 KiB). In the raw file, look for lines that didn't make it into the rules table: a typo in a directive (Dissalow) or a rule placed before the first User-agent is a common reason why "the file exists but doesn't work".

robots.txt syntax: directives and rule precedence

The file is made of groups. A group starts with one or more User-agent lines followed by Disallow (block crawling) and Allow (permit) rules. Sitemap isn't tied to a group and can go anywhere. Directive names are case-insensitive, paths are case-sensitive: /Catalog/ and /catalog/ are different rules. An empty Disallow: blocks nothing.

The * and $ wildcards

• * matches any sequence of characters: Disallow: /*?sort= blocks every URL with a sort parameter.
• $ marks the end of the URL: Disallow: /*.pdf$ blocks PDFs but not /file.pdf?v=2.

Which rule wins

Google doesn't read rules top to bottom. Among all rules matching a URL, the longest (most specific) one wins; on a tie, Allow wins.

User-agent: *
Disallow: /catalog/
Allow: /catalog/shoes/

Here /catalog/shoes/nike is crawlable and /catalog/bags/ is blocked.

User-agent groups don't add up

A crawler obeys only the most specific group that matches it. If there is a User-agent: Googlebot block, Googlebot ignores everything under User-agent: * — repeat those rules in its own block. This is one of the most common reasons why "blocked" sections still get crawled.

robots.txt examples you can adapt

WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/wp-sitemap.xml

Don't block /wp-content/ or /wp-includes/: they hold the CSS, JS and images Google needs to render the page. If an SEO plugin builds the sitemap, use its URL instead (for example /sitemap_index.xml).

Shopify, Wix, Squarespace and other hosted builders

Hosted platforms generate robots.txt for you; you control it through site and page settings (hide from search engines, exclude from sitemap) or a platform template, not by uploading a file. Check the live result after publishing — a "hide from search" toggle left on from development is a classic.

Online store: filters, sorting, UTM tags

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*utm_

Sitemap: https://example.com/sitemap.xml

A caveat on UTM: Google doesn't crawl a blocked URL, so it never sees the canonical on it. If tagged URLs attract many external links, keep them crawlable and point a canonical to the clean URL. Don't block filter landing pages that are meant to rank ("men's running shoes size 10").

Common robots.txt mistakes

• Disallow: / on the live site — the file was copied from staging. The checker flags this as a separate warning.
• Blocked CSS and JS (/assets/, /static/) — Google renders a "broken" page, which hurts how the mobile version is evaluated.
• Non-ASCII paths. Google accepts UTF-8, but percent-encoded paths (as they appear when copied from the address bar) are the safest choice. An internationalized domain in Sitemap must be in Punycode.
• BOM and wrong encoding. Google ignores a leading BOM, but other parsers may read the first line as garbage and lose the first group. Save the file as UTF-8 without BOM.
• Disallow instead of noindex. Blocking crawling doesn't remove a page from the index; it can stay in results without a description. Use noindex on a crawlable page instead — see noindex, nofollow and robots.
• Relative Sitemap URL (Sitemap: /sitemap.xml) — a full URL with protocol is required.

When robots.txt returns 5xx or a redirect

That makes a 5xx on robots.txt worse than having no file: crawling stops and crawl budget is wasted. Google usually caches the file for up to 24 hours, so fixes don't apply instantly.

Crawl-delay, Host, Clean-param, noindex: what Google ignores

• Crawl-delay — not supported by Google, which sets crawl rate from server responses. Bing honours it, which is why the checker warns about delays above 10 seconds.
• Host and Clean-param — Yandex-specific directives. Google skips them; use 301 redirects and canonical tags instead.
• Noindex in robots.txt — unsupported by Google since 2019.

These lines don't break the file — Google skips them and the rules for other crawlers keep working.

Testing robots.txt in Google Search Console

Google retired the old robots.txt Tester in 2023. Search Console now has a robots.txt report (Settings → robots.txt) showing which versions of the file Google fetched for each host, when, with what status and which lines it couldn't parse; you can also request a recrawl after an edit. To see whether a specific URL is blocked, use URL Inspection — it reports "Blocked by robots.txt". The full specification is in Google's documentation.

A practical workflow: validate the file here, build a new one with the robots.txt generator if needed, then make sure the sitemap from the Sitemap line opens and lists no blocked URLs using the sitemap analyzer.