Half your site may be outside Google's index — and you would not know
A page outside the index takes no part in search at all. Using our own data across nine sites, we look at why pages drop out, how to tell a crawling problem from a content problem, and in which of the two cases submission will not help.
A page outside the index takes no part in search at all. Not "ranks lower", not "gets less traffic" — it simply is not there: nobody will land on it from any query, no matter how many links you point at it or how much money you put into it. And yet almost nobody knows what share of their own site is in that state, because almost nobody checks the site as a whole.
Let us look at why pages fall out of the index, how to tell "not crawled" from "crawled and rejected" — these are fundamentally different problems with different cures — and what among it can actually be fixed.
How common is it
We looked at our own data. Our indexing section has accumulated 8,263 addresses across nine sites; among the checked pages the split is 4,258 indexed against 2,783 not — roughly 60 to 40.
But the average here is useless, and that is the main observation. The spread between sites turned out to be enormous.
Being straight about how this was measured and where the conclusion stops. The sample is small: five sites with a volume at which the figure means something. The method is indirect — we treat a page as indexed if it has impressions in Search Console over the last 16 months; a page can be in the index and have no impressions, in which case we wrongly count it as dropped. So this is not an industry statistic but an observation across our own sample.
The conclusion that follows from it, however, does not depend on sample size: you cannot predict your own number from someone else's benchmarks. The difference between 28% and 95% is the difference between "three quarters of the work does not exist for search" and "everything is fine", and the only way to know where you stand is to measure.
Two fundamentally different causes
When a page is missing from the index, that can mean two entirely different states, and confusing them is expensive.
First: Google has not crawled it. In the Search Console report this is the status Discovered — currently not indexed. Google states the reason plainly: the page was found, but crawling was rescheduled to avoid overloading the site. This is a problem of accessibility and crawl budget.
Second: Google crawled it and chose not to take it. The status Crawled — currently not indexed. The page was seen, read and not included in the index. And here is the key detail every indexing-service vendor stays quiet about:
Google on the status "Crawled — currently not indexed": the page may or may not be indexed later. No resubmission is needed.
Google Search Console Help, Page indexing reportWhich means that in this case resubmitting the URL will not help — not through the inspection tool, not through an API, not through any paid service. This is not a technical hiccup but a judgement: the content was not considered valuable enough to keep in the index. It is cured with content, not with submission.
The practical conclusion worth accepting before you pay anyone: submission helps with the first problem and is powerless against the second. That applies to our own indexing section too — it speeds up crawling, but it cannot make Google consider a page valuable.
The full list of causes
| What Search Console shows | What it means | What to do |
|---|---|---|
| Discovered — currently not indexed | Found but not crawled: Google rescheduled the crawl to avoid loading the site | Speed up crawling: sitemap, internal links to the page, external links, URL submission. Here submission genuinely works |
| Crawled — currently not indexed | Crawled and rejected. A judgement about content, not a failure | Submission does NOT help. Strengthen the content, merge thin pages, remove templated ones |
| Duplicate without user-selected canonical | Google considered the page a duplicate; you did not indicate a canonical | Set a canonical, or make the content genuinely different |
| Duplicate, Google chose different canonical than user | Your canonical was overridden in favour of another page | Compare both versions: they are usually too similar, and Google's choice is logical |
| Page with redirect | A non-canonical URL redirecting elsewhere — the target gets indexed, not it | Nothing, if the redirect is intended. Check that the target is the page you want |
| Excluded by 'noindex' tag | A direct ban on indexing in the markup or header | Remove noindex if the ban is not wanted. A common find: left over from development |
| Blocked by robots.txt | Crawling is forbidden by the robots.txt file | Lift the block. Important: robots.txt forbids CRAWLING, not indexing — to prevent indexing you need noindex |
| 404 / 403 / 401 / 5xx errors | The page is unavailable to the crawler | Fix availability. If it was removed on purpose — nothing |
| Soft 404 | The page returns a 200 code but looks like "nothing found" | Return an honest 404, or fill the page with content |
How to check your own site
Three ways, from the crudest to the most complete.
site:page-address in search. Found means indexed, not found means not. Fine for a quick check of a specific address, but it gives no picture of the site.Whichever you choose, look at the share and not the absolute number. "1,200 pages indexed" says nothing until you know how many there are in total.
What usually turns out to be the cause
From the experience of picking apart other people's sites, the list of suspects is fairly short.
noindex left over from development, or a section closed in robots.txt that should have been opened long ago. Takes a minute to check and shows up more often than you would think.What not to do
noindex, and that requires the page to be crawlable.The order of operations
If you are starting from scratch, a sensible sequence looks like this.
First measure the share across the whole site — without that everything else is guesswork. Then open the Page indexing report and see which status dominates: "discovered" or "crawled". That is the fork that determines everything downstream.
If "discovered" dominates, the problem is accessibility and crawling. Here the sitemap, internal linking, external links and direct URL submission all work, and the result is usually visible within weeks.
If "crawled" dominates, the problem is the content, and submission will not help. You will have to look at what exactly Google did not consider worthy of the index and decide: strengthen, merge or remove.
And only after that does it make sense to invest in promoting specific pages. Links to a page outside the index are money spent into a void.
Collects pages from the sitemap and Search Console and checks them in a batch. Checking is free, with no limit on how many.
Frequently asked questions
How do I find out how many of my site's pages are in Google's index?
The full picture comes from the Page indexing report in Search Console — it shows both the count and the reasons for each category. An individual page can be checked with a site:page-address query. To assess the whole site you need a bulk check of a list of addresses.
What is the difference between "discovered" and "crawled" when a page is not in the index?
"Discovered — currently not indexed" means Google found the page but has not crawled it yet: a matter of accessibility and crawl budget. "Crawled — currently not indexed" means the page was crawled and not included in the index: a judgement about the content.
Will resubmitting a page for indexing help?
It depends on the status. With "discovered but not crawled", submission speeds up crawling and works. With "crawled — currently not indexed", Google states directly that resubmission is not required — there the cure is content, not submission.
What share of pages in the index counts as normal?
There is no universal norm. Across our sample of five sites the share ranged from 28% to 95% at comparable volumes, so orienting yourself by someone else's numbers is pointless — your own value can only be measured. And 100% is not the aim either: utility pages, correct duplicates and pagination should not be in the index.
Why is a page not being indexed even though there are no errors?
The most common reason is that the content was not considered valuable enough: thin pages, templated cards differing by one parameter, automatically generated archives. The second most common is that the page has no internal links and sits on the periphery of the site.
Should I block unnecessary pages in robots.txt?
Not for excluding them from the index. Robots.txt forbids crawling, not indexing: a blocked page can remain in the index via external links, only without content. Exclusion requires a noindex tag, and for Google to see it the page must be crawlable.
Sources
Google Search Console Help, Page indexing report: the list of statuses, the wording of each reason and the recommended action.
PromoPilot's own data, August 2026: 8,263 addresses across 9 sites; among the checked pages, 4,258 indexed and 2,783 outside the index. Indexation was determined by the presence of impressions in Search Console over 16 months.
