Audit: Indexing Issues and What They Mean
The 'Indexing and Crawling' category answers a single question: will the page appear in search. Here, six out of twelve audit errors exist — if the page is unavailable or closed, other remarks about it make no sense. This also includes checks for robots.txt and access for AI crawlers.
Errors: Fix First
| Code in Report | What Happened | What to Do |
|---|---|---|
| Page Returns 4xx | The address does not exist, but there are links to it | Restore the page or set a 301 to the current address and remove the links |
| Server Error 5xx | The server could not return the page | Check the logs: usually it's a database, limits, or a code error |
| Page Unavailable | Timeout or connection drop | Check the stability of the hosting and response time |
| Redirect Loop | The address redirects to itself in a loop | Break the chain: usually a conflict of www/https rules and CMS |
| Multiple Canonicals | There are two or more canonical addresses on the page | Keep one — otherwise, the search will choose one itself |
| Site Content Rendered by JavaScript | There is almost no text and links in the original HTML — everything appears only after executing scripts | Enable server-side rendering (SSR) or prerendering, make links regular <a href> tags |
Warnings
- Page Closed with noindex — sometimes intentionally, but more often it's a forgotten restriction after development. Check if this page is really needed in search.
- Closed in robots.txt — the bot will not crawl the page. Remember: this is not the same as removal from the index, see analysis of three directives.
- Redirect Chain — more than one step: speed and part of the weight are lost.
- Meta Refresh — an outdated method of redirection, search understands it worse than a regular redirect.
- Hreflang without self-reference — language version markup does not work, see hreflang check.
- robots.txt Returns HTML Page — at the address /robots.txt the server returns a site page instead of rules. For search engines, robots.txt effectively does not exist, and the sitemap cannot be found from it. This is a common issue for JavaScript sites that respond with the homepage to any address.
- AI Crawlers Closed in robots.txt — bots like ChatGPT, Claude, Perplexity, or Google AI cannot read the site, meaning they will not quote it in their responses. Details of the issues are listed for the closed bots.
Remarks
No canonical and canonical to another address — the second is particularly insidious: the page is excluded from the index in favor of the specified one. Internal Redirect — links within the site lead to an address that redirects; it's better to link directly to the final one. Access Restricted — hosting protection has blocked our crawler: the page is likely working, but check manually. Sitemap.xml Not Found — see sitemap check. No robots.txt — there is nothing at the address /robots.txt, search engines do not see crawl rules and links to the sitemap; create a file with rules and a line Sitemap: with the sitemap address.
JavaScript Sites
If there is almost no text and links in the main code, the audit opens pages in a real browser and checks what the visitor sees, and a banner appears above the report saying 'Site Content Rendered by JavaScript — we checked it in the browser'. The list of problems in this case is complete, but the error 'Site Content Rendered by JavaScript' remains: Google renders such pages with a delay and not always, while AI bots usually see an empty page. If checking in the browser was not possible, the banner will be without words about the browser, and the number of found pages will be low: this is how the site is seen by search engines without rendering.
Access for AI Crawlers and llms.txt
The audit reads robots.txt and checks if the site is open to bots like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, and Applebot-Extended. If at least one is closed, a warning appears 'AI Crawlers Closed in robots.txt'. Whether to close bots or not is your decision, but a closed bot will not quote the site.
Additionally, llms.txt is checked — a brief sitemap for AI assistants in Markdown format: what the company is about, which sections and pages are main. If the file is missing, a remark 'No llms.txt' appears; in the report, it is categorized under 'Content'. This is a remark, not an error: the file is useful but not mandatory. How this relates to quoting in AI responses is discussed in the section 'Visibility in AI Search' of the article how to read the audit report.
How to Check a Specific Page
The audit shows the picture for the site, but for a single page, it's faster to use indexing checker and response headers — the restriction may be returned by the server, and it may not be visible in the page code.