Why a page is not in Google, in the order to check it

Crawled, indexed and ranked are three different problems

Most confusion about indexing dissolves once these are separated. Crawled means a search engine fetched the URL. Indexed means it decided to store the page and consider it for results. Ranked means it shows it for a particular query. A page can be crawled and not indexed, and indexed and never ranked, and the fix is completely different in each case.

This page is about the first two, because they are mechanical and you control them entirely. If a page is indexed and simply not ranking, no technical checklist will help - that is a content and competition question, and every hour spent re-checking canonicals is an hour not spent on the answer.

The fastest way to tell which problem you have: search for the exact URL, or use the URL Inspection tool in Google Search Console. If the page is known but excluded, the reason is stated there in one line, and everything below is how to fix each of those lines.

1. A noindex you did not mean to ship

This is the single most common cause, and it is almost always the same story: a staging environment carried a site-wide noindex, and the deploy that took it live did not remove it. It can arrive in two places, and a checklist that looks at only one of them misses half the cases.

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

The header form is the one people miss, because it is invisible in the page source and often set at a server or CDN level by somebody who left. Check the response headers, not just the HTML, and check them on a page deep in the site rather than only on the homepage.

2. robots.txt blocking what you wanted crawled

robots.txt controls crawling, not indexing, and the difference matters more than it sounds. A URL that is disallowed can still appear in results - with no description, because the crawler was never allowed to read the page to write one.

That leads to the most common robots.txt mistake there is: blocking a page in robots.txt in order to keep it out of the index. It does the opposite of what you want. If the page must not be indexed, it has to be crawlable so the crawler can see the noindex directive telling it so.

• Disallow: / left over from a staging config - the whole site, invisible, usually noticed weeks later
• Blocking /assets or /static, which stops the CSS and JS from being fetched and makes the rendered page look broken to the crawler
• A Disallow that also matches URLs you wanted, because the rule is a prefix and not a path
• An unreachable robots.txt - a 401, 403, 429 or 5xx is treated as 'do not crawl anything', while a 404 correctly means 'no restrictions'

3. Status codes and redirect chains

A page has to return 200 to be indexed. Two failure modes are worth checking specifically because both look fine to a human.

The first is a soft 404: a page that says 'not found' in its content while returning 200. Search engines detect these and drop them, and you get no error to investigate because nothing errored. Anything that renders a not-found state must return an actual 404.

The second is redirect chains. Each hop loses a little and costs crawl budget, and a chain long enough will simply be abandoned. The classic accumulation is http → https → www → trailing slash → the final URL: four hops that could be one, all internal, all fixable by pointing links at the destination rather than at the first link in the chain.

Fix internal links to point at the canonical form directly. Redirects are for other people's links and old bookmarks; a site redirecting its own navigation is paying the cost on every crawl of every page.

4. Canonicals pointing somewhere unhelpful

A canonical link tells a search engine which URL owns a piece of content when several serve it. Get it wrong and you have told the engine, in writing, not to index the page you care about.

• Every page canonicalising to the homepage - a template bug, and it removes the whole site from results except one URL
• A canonical pointing at a URL that redirects, 404s, or is itself noindexed
• A canonical on a staging or preview host, pointing production content at an origin nobody can reach
• Missing canonicals on a site that serves the same content at /page and /page/, and with and without www - all four are separate URLs to a crawler

Pick one form of every URL and make the site agree with itself: one host, one protocol, one trailing-slash convention. Then have the canonical say so on every page, including the one it points at.

5. Content that only exists after JavaScript runs

Google does render JavaScript, so this is no longer the automatic disaster it once was. It is still a real risk, for two reasons that have nothing to do with whether rendering works.

Rendering happens in a second pass, sometimes days after the initial crawl, and the budget for it is finite. And every other consumer of your pages - other search engines, link previews in Slack and LinkedIn, and most of the crawlers that feed AI answers - reads the served HTML and stops.

The test takes ten seconds. Fetch the page the way a crawler does and look for your own content:

curl -s https://example.com/page | grep -i "a phrase from the page"

If the phrase is not in the response, the served HTML is an empty shell. Server-side rendering or prerendering for the routes that matter is the fix; client-side rendering is fine for anything behind a login, which is not indexed anyway.

6. Nothing links to it

A page with no internal links pointing at it is a page a crawler has no route to. Sitemaps help it get discovered, but a sitemap is a hint about what exists - not an argument that a page matters. Internal links are the argument.

• The page should be reachable from the homepage in a small number of clicks; deep in a pagination tail is where pages go to be forgotten
• The links pointing at it should have descriptive text, because that text is one of the strongest signals about what the page is for
• It should be in the XML sitemap, and the sitemap should not list URLs that 404, redirect or are noindexed - a sitemap full of those teaches a crawler to trust it less

7. Crawled, and deliberately not indexed

The hardest case: everything above is correct, the page is fetched, and Search Console says 'Crawled - currently not indexed'. That is not a technical fault. It is a judgement that the page is not worth storing, and the usual reasons are thin content, near-duplicates of other pages on the same site, or a page generated by a template that produces hundreds of variations of the same thing.

No technical fix exists for it, and this is where a technical audit has to say so rather than keep offering things to check. The answer is fewer, better pages: consolidate near-duplicates, remove the ones that exist only to target a keyword, and make the survivors worth the storage.

A checklist that never says 'this one is not a technical problem' will send you round the same seven items forever. Knowing which of your pages are in this state, and accepting it, is more valuable than the checklist.