How to Check robots.txt: Test the File and Specific URLs, Common Mistakes

Checking robots.txt comes down to three questions: does the file load at https://your-site/robots.txt with a 200 status, does it contain rules that block sections you need, and are any important URLs caught by a Disallow. robots.txt controls crawling, and one wrong line can block the whole site — so check the file after every change. You can analyze your file with the robots.txt checker and build a new one from scratch with the generator.

How to check robots.txt, step by step

1. Open the file in a browser: https://example.com/robots.txt. It must sit at the root of the host. Rules apply only to the protocol, host and port where the file lives: shop.example.com has its own robots.txt, separate from example.com.
2. Check the status code: curl -I https://example.com/robots.txt. You want 200. A redirect is fine, but Google follows at most five hops. What other codes mean is in the table below.
3. Run the file through a parser. The robots.txt checker fetches the file, groups the rules by User-agent, and shows Allow and Disallow, Sitemap, Crawl-delay and the raw text. It warns you if Disallow: / applies to all bots, if there is no Sitemap line, if Crawl-delay is above 10 or the file is too large, and gives a 0–100 score.
4. Test your key URLs against the rules: the homepage, categories, the pages you promote, CSS and JS files. How to do it by hand is in the next section.
5. Open the robots.txt report in Search Console (Settings → robots.txt). It shows which robots.txt files Google found for the top 20 hosts of your site, when it last fetched them, their status, size and any parsing issues. After an urgent fix you can request a recrawl of the file from the same report. The report is available for Domain properties and URL-prefix properties without a path. The old standalone robots.txt Tester is gone — this report replaced it.
6. Check a specific page with URL Inspection in Search Console: it tells you whether crawling is allowed and, if not, what blocks it.

What our checker does not do: it does not test a specific URL against the rules, and it does not know how a particular crawler interprets them. It shows the file as any visitor sees it and catches the obvious mistakes. Test individual URLs with the method below or in Search Console.

What the tool checks

• Fetches the file from the domain root over HTTPS — https://domain/robots.txt, following redirects — and tells you if the file is missing (404), the server returned a 4xx or 5xx error, or the file could not be loaded.
• Groups the rules by User-agent: which paths are blocked (Disallow) or allowed (Allow) and for which bots. It also counts lines and rules.
• Lists the sitemap links (Sitemap) and the Crawl-delay value.
• Warns about obvious problems: no User-agent group at all; Disallow: / for all bots; no Sitemap link; Crawl-delay above 10; a file longer than 500 lines or larger than 512 KB.
• Gives a score from 0 to 100 and shows the raw file.

What the tool does not check: syntax and typos — a line with a misspelled directive is simply skipped; whether CSS, JS or specific URLs are blocked — test those by hand using the rules below or in Search Console.

Typical mistakes

How to test whether a URL is blocked

1. Find the group that applies. A crawler obeys only one group — the one whose User-agent matches its name most specifically. If the file has a User-agent: Googlebot group, the rules under User-agent: * no longer apply to Googlebot. Google combines multiple groups for the same User-agent.
2. List the rules that match the start of the URL path. Matching is case-sensitive, and the query string after ? is part of the URL.
3. The longest matching rule wins. If a matching Allow and Disallow are the same length, Google uses the least restrictive one — Allow.
4. No match at all means the URL is allowed.

Two special characters work in paths: * matches any sequence of characters, including none, and $ marks the end of the URL.

robots.txt status codes and Google's limits

The non-obvious conclusion: a robots.txt that returns 500 by mistake is more dangerous than no file at all — a 404 means "crawl everything", while a 5xx stops crawling. Two more limits from Google's robots.txt documentation: Google reads at most 500 KiB of the file and ignores the rest, and it usually caches the contents for up to 24 hours, so edits do not take effect instantly.

Directives not to rely on

Some lines found in others' files are ignored by modern robots or understood differently: crawl delay, specifying the main mirror, non-standard address patterns. Google officially supports a limited set of directives; everything else either does not work or is interpreted differently by various search engines. Therefore, the shorter and simpler the file, the more predictable the result.

robots.txt and AI bots

AI services crawl with their own bots, and you block them in robots.txt the same way — with separate User-agent groups: GPTBot, ClaudeBot, PerplexityBot and others. Google-Extended is not a separate crawler but a token: it controls whether your content is used for Google's AI models and has no effect on regular Search. Do not mix them up: blocking training and blocking AI search are different bots, and blocking everything can take you out of AI assistants' answers. Ready-made blocks and what each one costs you are in how to block AI bots in robots.txt, and the robots.txt generator builds a file with the groups you choose.

Minimum working file

For most sites, it is enough to open everything, close service sections, the trash, and results of internal search, and then specify the sitemap. Everything else is as needed. Check after changes that the necessary pages remain accessible: index checker and in Search Console.

Related checks

At the same time, it is worth looking at the sitemap and the difference between noindex, nofollow, and robots.txt — these three things are often confused.