Checking robots.txt: what is unnecessarily closed and not closed
robots.txt manages site crawling: which sections robots should view and which ones they should not. An error in one line can close the entire site — therefore, the file should be checked after each change. You can analyze your file with robots.txt checker, and create a new one from scratch with generator.
What the tool checks
- Is the file accessible at
/robots.txtand is it returned with a 200 code. - Are there any syntax errors or typos in the directives.
- Which sections are closed and for which robots.
- Is a sitemap specified.
- Are any necessary pages and style files with scripts accidentally closed.
Typical mistakes
| Error | Consequence |
|---|---|
Disallow: / for all robots | The site is completely closed to crawling — a classic mistake after moving from a test server |
| CSS and JS are closed | The robot sees the page differently than the user and evaluates it worse |
| Attempting to remove a page from the index via robots.txt | Does not work: disallowing crawling does not equal removal from the index, for that a noindex is needed |
| No link to the sitemap | The robot takes longer to find new pages |
| Different files on www and without www | Behavior depends on which version is crawled |
Directives not to rely on
Some lines found in others' files are ignored by modern robots or understood differently: crawl delay, specifying the main mirror, non-standard address patterns. Google officially supports a limited set of directives; everything else either does not work or is interpreted differently by various search engines. Therefore, the shorter and simpler the file, the more predictable the result.
Minimum working file
For most sites, it is enough to open everything, close service sections, the trash, and results of internal search, and then specify the sitemap. Everything else is as needed. Check after changes that the necessary pages remain accessible: index checker and in Search Console.
Related checks
At the same time, it is worth looking at the sitemap and the difference between noindex, nofollow, and robots.txt — these three things are often confused.