A material study in bright daylight. A half circle fan of clear turquoise glass blades spreads from a small point at the lower right, bound along its outer edge by a polished pearl silver rim.
  1. Home
  2. Insights
  3. Search and AI Visibility
  4. Not appearing on Google

Technical explainer

Why a site does notappear on Google.

Three different problems look identical from outside. The page is not in the index. The page is in the index and ranks badly. Or the page is in the index under a different address. They have different causes, different fixes and very different costs, and checking them in the wrong order wastes months.

Every check below can be run by the person who owns the site, without a tool and without a supplier. The first three take about five minutes and account for most cases that turn out to be fixable in an afternoon. Written 23 September 2026.

HUREAL / Material studies

First, which problem

Three problemsthat look identical.

Almost every conversation about a site not appearing starts in the wrong place, because the symptom is the same in three different situations and the three have almost nothing in common.

Three problems, how to tell them apart, and what each one costs to fix
The problem How you tell What it usually costs to fix
The page is not in the index Searching for the exact address returns nothing. Often minutes. This is where the cheap, catastrophic causes live.
The page is in the index and ranks badly Searching for the exact address returns it. Searching for the subject does not. Weeks to months. This is a content, structure and authority problem.
The page is in the index under a different address A different URL appears where you expected this one, or a version with parameters, or the same page on another protocol or subdomain. Days. Canonical and duplication work, and it is worth doing before anything else.

Scroll the table sideways to read it.

The distinction costs about a minute to establish and it changes the whole investigation. Searching for the exact address of the page, in quotation marks, answers it. So does listing what the search engine holds for the domain, using the site operator. Do that before reading any further, because two thirds of this article will not apply to you.

The sequence

Nine checks,in this order.

The order is not arbitrary. Each check is cheaper than the one after it and more completely fatal, so working down the list finds the expensive, silly causes before anybody spends a quarter on the expensive, legitimate ones.

  1. Does the page return successfully, from outside your network

    Not from a browser that has a session, a cookie and a cached copy. From outside, cleanly. A page that intermittently times out or returns a server error during a crawl is a page that gets crawled less and eventually dropped, and intermittent failures are invisible to anybody who only ever loads it once.

  2. The robots file

    The single most common catastrophic cause, and it is almost always a line left over from a staging environment that was copied to production along with everything else. Open the file directly and read it. A disallow on the root blocks the entire site, and it has been sitting there since launch on more sites than anybody would like to admit.

  3. Directives that say not to index

    Two places, and only one of them is visible in the page source. A meta tag in the head is easy to find. The same instruction delivered as an HTTP response header is not in the source at all, which is why it survives for months while everybody looks at the markup. Check the headers, not just the page.

  4. Anything that requires a credential

    Basic authentication, an IP allowlist, a maintenance mode, a geographic block, or a bot management product that challenges anything without a browser fingerprint. All of these return something other than the page to a crawler while working perfectly for every human who tests it.

  5. The canonical tag

    A canonical pointing at a staging domain is the classic launch failure. A whole site canonicalised to its own homepage is the classic template failure. Both tell the index that this page is a copy of something else and should be ignored in favour of it, and both are a one line fix once seen.

  6. What is actually in the response

    Fetch the page without a browser and read what comes back. If the content only exists after scripts run, indexing becomes slower and less reliable, and on a large site a meaningful share of pages simply will not make it.

  7. Whether anything links to it

    A page with no internal links pointing at it is reachable only through the sitemap, and a sitemap is a suggestion rather than an instruction. Orphan pages are created constantly by accident: a section is redesigned, a navigation item is dropped, and ten pages become unreachable while still existing.

  8. The sitemap, and what is in it

    Present, valid, referenced from the robots file, and listing the canonical version of each address. A sitemap full of URLs that redirect, or that are blocked, or that carry parameters, is worse than a smaller correct one, because it spends the crawler's attention on addresses that go nowhere.

  9. Near duplicates competing with each other

    The same content reachable on two protocols, with and without a subdomain prefix, with and without a trailing slash, with tracking parameters, in a printable version, and through a faceted navigation that produces one address per combination of filters. Each variation splits the signals and dilutes all of them.

Only after all nine is it reasonable to ask whether the page is any good for the query, which is the expensive question and the one everybody starts with. The order exists because the fixes cost radically different amounts: a line in a text file is two minutes, a duplication problem is a week, and a quality problem is a programme of work. Starting at the bottom is how a quarter gets spent writing content for pages a crawler was never allowed to fetch.

The report

The index reportalready knows.

Most of the sequence above can be skipped, because the search engine publishes its own reasons. The page indexing report in Search Console lists every address it knows about and groups them by why they are or are not indexed. Two of those groups are routinely misread, and the misreading sends teams in exactly the wrong direction.

What the index report is actually saying, and what it means for the fix
What the report says What happened What to do about it
Blocked by the robots file The crawler was told not to fetch it. Fix the file. This is the two minute case.
Excluded by a no index directive It was fetched and explicitly told not to index. Find the directive. Check the response headers as well as the markup.
Discovered, not yet crawled The address is known and has not been fetched. Usually a signal about crawl priority. More internal links to it, and fewer worthless addresses competing for attention.
Crawled, not indexed It was fetched, read, and judged not worth adding. Not a technical problem, and the most commonly misdiagnosed status on this list. Requesting indexing repeatedly does nothing. The page has to be worth indexing.
Duplicate, with no canonical chosen by you Several addresses look the same and none of them declared which is primary. Declare a canonical, and consider whether the variants should exist at all.
An alternate page with a proper canonical tag Working as intended. This address points at another one, which is indexed. Nothing, unless the canonical points somewhere you did not intend, which happens more often than it should.
A page with a redirect The address forwards elsewhere. Nothing, unless it is in the sitemap, in which case the sitemap is out of date.
Soft 404 The page returned a success code while looking like an error or an empty result. Return a proper not found code, or put real content on it. Empty category and search result pages are the usual source.

Scroll the table sideways to read it.

The distinction between a page that was discovered and not crawled, and one that was crawled and not indexed, is the most useful single thing in this report. The first is a question about attention: the crawler has finite time for your site and is spending it elsewhere. The second is a judgement that has already been made about the page itself, and no amount of resubmission changes a judgement.

The large site case

Filters can generatemore pages than you sell.

On a catalogue of any size, the most common reason products are missing from the index has nothing to do with the products. It is that the filters have generated an enormous number of addresses, and the crawler is spending its attention on them.

Four filters with a handful of options each, combinable in any order, produce thousands of distinct addresses. Add a sort order and a page number and it is tens of thousands. Almost all of them return nearly the same products in a different arrangement, which means they are near duplicates of each other, and every one of them is an address the crawler may fetch instead of fetching a product page it has not seen.

The fix is a set of decisions rather than a setting. Which filter combinations are worth having as their own indexable pages, because somebody actually searches for that combination. Which should exist for a visitor but declare a canonical pointing at the unfiltered category. Which should not be crawlable at all. And what the sitemap should contain, which is the canonical set and nothing else.

The diagnostic that shows this immediately. Compare the number of addresses the index report knows about with the number of pages you believe you have. If the first is several times the second, the crawler is working through a space you did not intend to create, and no amount of work on individual product pages will be noticed until that is resolved.

The other kind of cause

Three causesno configuration fixes.

The content already exists elsewhere

A catalogue built on the descriptions the manufacturer supplies is a catalogue of text that appears on every other distributor's site and on the manufacturer's own. There is no technical error and nothing to configure. The index has many copies of that text and a reason to prefer one of the others.

The fix is the expensive one and it is also the only one: original content on the pages that matter. Specifications in your own words, the application notes your sales team gives on the phone, what it fits, what it does not fit, what people ask before buying it.

The page answers nothing specific

A page about a service that could describe any supplier of that service has nothing to be retrieved for. It is not penalised, it is simply not the best answer to any particular question, which produces exactly the same outcome.

The test is whether the page contains anything only somebody who does this work would know, and whether it names the thing a buyer actually types rather than the thing the industry calls it.

The third cause is the simplest and the most common on a new site: nothing links to the page, from anywhere, and a page with no route to it is a page with no reason to be found. Internal links first, because they are free and entirely under your control.

One honest caveat

On a new site,some of this is just time.

A domain with no history has nothing for an index to go on. Pages get discovered, some get indexed, and the shape of what the site is about resolves over months rather than weeks. During that period a page can be indexed, drop out, and return, and none of that is necessarily a fault.

What is worth separating is the part that is time from the part that is a cause. The nine checks above are all answerable on day one and none of them improves with waiting. If every one of them passes and pages are still being discovered and not crawled after a few months, that is a signal about attention and authority rather than about configuration, and the levers for it are internal links, a reason for other sites to link, and pages worth indexing.

The mistake that costs the most is the opposite one: deciding that a site is in a waiting period when it is in fact returning a directive that says not to index. That state also looks like nothing happening, it also resolves to an explanation involving patience, and it can persist for a year without anybody opening the file.

Questions

Questions peopleactually ask.

  • How long should indexing take for a new page?

    Days to weeks on an established site, and longer on a new domain where the index has little history to go on. What is not normal is months with no change on a site that is otherwise indexed, and that pattern usually points at a specific cause rather than at patience being required.

  • We submitted the sitemap. Why is the page still not indexed?

    A sitemap is a suggestion about what exists, not an instruction to index it. It helps a crawler discover pages it might otherwise not reach, and it does nothing about a page that has been discovered and judged not worth indexing. If the report says a page was crawled and not indexed, the sitemap is not the lever.

  • Should we request indexing for every page by hand?

    No. It is useful for a handful of important pages after a fix, as a way of asking for a recrawl sooner. It does not scale, it does not change the underlying judgement, and using it as a routine process usually means something structural is being worked around rather than found.

  • A page used to be indexed and now it is not. What happened?

    Most often one of four things: something changed at the technical level, such as a directive added by a plugin or a platform update, the page was consolidated with a near duplicate, the page lost the internal links that pointed at it, or the content was judged to add nothing. The index report distinguishes those, which is why it is the first place to look rather than the last.

  • Can we pay to have pages indexed faster?

    No. Advertising buys placement in the advertising slots, which are labelled and separate, and it does not affect the index. Anybody offering paid indexing as a service is offering something that is not for sale.

Read next

Three that followfrom this one.

The service behind this article

Every check hereis in the audit.

This sequence is the technical half of a Search and AI Visibility engagement, and it is run against a named site rather than described. The output is which check failed, on which URLs, and what the fix costs, in that order.

A performance audit runs this sequence against your site and reports where it stops. The robots file, the response headers, canonical behaviour, what is in the HTML before any script runs, the internal link graph and the duplication picture. It is a document about your company, and it is yours whether or not anything follows it.

Request a performance audit

More on this subject in the Search and AI Visibility index, and everything else at Insights.

A material study photographed in bright daylight. A tall fan of clear turquoise glass fins rises on the right of the frame with a polished pearl silver ribbon curving through it, standing in a shallow film of still water. The left of the frame is empty pale mint.

Software you own from day one.

HUREAL / Material studies