Site indexing in Yandex and Google: how to check it and what blocks it

How a search engine discovers and indexes pages

Discovery, crawling and indexing are three different stages

A page’s path into search consists of three stages:

1. Discovery. The search engine learns the page’s address: from links on pages it already knows about, or from a Sitemap file submitted by the site owner. Yandex also lists Yandex Metrica data among its sources.
2. Crawling. A crawler downloads the page. The Yandex robot decides for itself which sites to visit, how often, and how many pages to crawl. Google can fail to crawl a page, for example, because of server or network problems, or if access is blocked by robots.txt rules.
3. Indexing. The engine analyzes the text, images and video on the page and stores information about it in the index. At this stage, Google checks whether the page duplicates another one and selects the primary (canonical) address.

Distinguishing between the stages matters because a failure looks different at each one. For example, Search Console has separate statuses for a page that Google has found but usually hasn’t crawled yet (“Discovered – currently not indexed”), and for a page that it crawled but didn’t add to the index (“Crawled – currently not indexed”).

None of these stages is guaranteed. Google states outright that it doesn’t guarantee crawling, indexing or displaying a site. Yandex, too, doesn’t include every page in search.

Why “indexed” doesn’t mean “ranking well”

Indexing is admission into search, not a place in the results. The order of results is determined by ranking: for a user’s query, the engine selects pages from the index and weighs hundreds of factors in the process.

So “the page isn’t indexed” and “the page is indexed, but ranks low” are different problems. The first is solved with the technical diagnostics this article covers. The second requires working on content, search intent and competition in the results.

How to check indexing

Yandex Webmaster: pages in search and excluded pages

The main report is “Pages in search” (section “Indexing” → “Pages in search”). It has three tabs: “Recent changes,” “All pages” and “Excluded pages.” For each page you can see the date of the robot’s last visit and its status, and on the “Recent changes” tab also the HTTP code the robot received.

The status shows whether the page is in search and, if not, why. Example statuses in the interface: “In search,” “Low-value or low-demand page,” “Duplicate,” “Non-canonical,” “Blocked by the noindex element,” “Blocked in robots.txt (page).” In the exported file these correspond to the codes SEARCHABLE, LOW_DEMAND, DUPLICATE, NOT_CANONICAL, META_NO_INDEX and ROBOTS_TXT_ERROR.

Excluded pages aren’t necessarily a problem. According to Yandex’s help center, the presence and number of excluded pages shouldn’t affect the site’s position in search results. What matters is something else: whether any pages that should be in search are among the excluded ones.

For a specific address there’s the “Page check” tool (“Indexing” → “Page check”). It shows the response code the robot receives, how the page’s content looks to the robot, and the page’s status: in search or excluded.

Google Search Console: the indexing report and URL inspection

In Search Console, the overall picture comes from the “Page indexing report.” It splits pages into indexed and not indexed, and gives a reason for each unindexed one. Google notes that on most sites some pages remain unindexed; the goal is for all key pages to be processed while copies and alternate versions aren’t indexed.

A specific address is checked in the “URL Inspection tool.” It shows the page’s status in the index and the field “Google-selected canonical” — the address Google considers primary among similar versions. The “Test live URL” button runs a check of the page’s current version. A positive result from this check doesn’t guarantee that the page will appear in search results.

Why site: doesn’t replace Yandex Webmaster and Search Console

A query like site:example.ru is handy for a quick sanity check, but you shouldn’t base decisions on it. Google’s documentation states that the site: operator doesn’t always return every indexed URL, and recommends checking a missing page with the URL Inspection tool.

Yandex Webmaster’s help center has no official statement about the accuracy of site:. To check a specific page, the help center suggests searching for its address, including with the url: operator, and checking the number of indexed pages in Webmaster.

Bottom line: the operator helps you quickly check a hunch, but draw conclusions about indexing status from Webmaster and Search Console data.

Why a page isn’t indexed: a diagnostic tree

It’s easiest to check the causes in order: first whatever blocks access completely (robots.txt, noindex, response codes), then canonical and duplicates, then how the search engine finds the page, and finally an assessment of its content.

Cause	How to check	What to do
Crawling is blocked in robots.txt	Webmaster: the status “Blocked in robots.txt (page)” in “Pages in search.” Search Console: the reason “Blocked by robots.txt”	If the page should be in search, remove the Disallow rule that blocks it
noindex in meta robots or in the X-Robots-Tag HTTP header	Webmaster: the status “Blocked by the noindex element.” Search Console: the reason “Excluded by ‘noindex’ tag”	Remove noindex from pages that should be indexed. Check that the page isn’t blocked in robots.txt, otherwise the robot won’t see the change
Canonical points to a different page; the page is a duplicate	Webmaster: the statuses “Duplicate” and “Non-canonical” among excluded pages. Search Console: “Alternate page with proper canonical tag,” “Duplicate, Google chose different canonical than user,” the field “Google-selected canonical”	A standalone page’s canonical should point to itself. Merge genuine duplicates, or keep only one primary version
Response code other than 200: 404, 5xx, a redirect	Webmaster: the response code in “Page check” and on the “Recent changes” tab of the “Pages in search” report. Search Console: “Not found (404),” “Server error (5xx),” “Page with redirect,” “Redirect error,” “Soft 404”	A page you want indexed as a standalone URL must open normally and return a 200 code. Point a redirect straight to the final address. Fix server errors on the hosting or application side
The page is hard to find	Whether the address is in the Sitemap file; whether other pages on the site link to it	Add internal links from related pages and include the address in the Sitemap
The page is low-value or very similar to others	Webmaster: the status “Low-value or low-demand page.” Search Console: “Crawled – currently not indexed” — Google doesn’t name a specific reason, so assess the content yourself; Google doesn’t require resubmitting such a URL for crawling	Decide whether the page needs to be in search. If it does, make its content self-sufficient and useful for a specific query. Merge similar pages; close off utility pages with noindex

If the table doesn’t pinpoint the cause, or many pages are affected at once, you need to check templates, utility files and server responses across the whole site. That’s a technical SEO task — more on how checking indexing and technical errors works.

Blocking crawling in robots.txt

robots.txt is a text file with indexing parameters for a site, aimed at search robots. The Disallow directive blocks crawling of sections or individual pages.

robots.txt controls crawling, not the index. Google says the file is meant primarily to manage robot traffic and isn’t designed to keep material out of search results: pages blocked in it can still be shown, just without a description. Yandex’s help center likewise says pages restricted in robots.txt can still appear in search, and recommends noindex for removing a page from search.

Two conclusions follow. If a page should be in search, check that it isn’t blocked in robots.txt. If a page needs to be removed from search, robots.txt alone isn’t enough. The file’s syntax and how robots.txt rules work are covered separately.

noindex in meta robots or an HTTP header

noindex is a directive that blocks a page from being indexed. It’s set in the robots meta tag in the page’s code or in the X-Robots-Tag HTTP header. According to Yandex’s help center, a page with noindex won’t appear in search results. When Google finds the rule while crawling, it excludes the page from search results — even if other sites link to it.

There’s an important interaction with robots.txt. Google says: if robots.txt blocks the robot’s access to a page, it can’t process the page’s code and won’t discover the noindex rule, so the page may still appear in search results. Yandex puts it this way: if a page is blocked in robots.txt, the meta tag or header directive has no effect. So blocking a page in both robots.txt and via noindex at the same time is pointless: the robot won’t read the second directive.

Canonical points to a different page; duplicates

The rel=»canonical» attribute tells a search engine which address to treat as primary among similar versions of a page. Both engines treat it as a hint rather than a binding rule. Google states outright that it recommends ways to specify a canonical address, but they aren’t mandatory: without them, it will determine the most suitable version on its own.

• Google: even if you specify a canonical page, Google may choose a different one for one reason or another. It crawls copies and variants of the primary page less often.
• Yandex: the robot treats the canonical address as a recommendation and may ignore it — for example, if the canonical page’s content differs significantly from the non-canonical one.

A common mistake is a standalone page with canonical pointing to a different address, for example the homepage or a section’s first page. If the search engine accepts that signal, the page can drop out of search as non-canonical. You can see which address Google chose as primary in the “Google-selected canonical” field of the URL Inspection tool. For how to set the primary address and check the search engine’s choice, see the article more on canonical tags and duplicates.

Response codes, redirects, server errors

An HTTP response code is a three-digit number a server uses to reply to a robot’s request. A page you want indexed as a standalone URL must open normally and return a 200 code.

• Redirects (3xx). The Yandex robot will index the page a redirect points to, if it’s accessible. By default, Google follows a limited number of hops in a chain and processes only the final address.
• 4xx errors. Google doesn’t use content from addresses that return a 4xx code, and removes already-indexed pages like that from the index. For Yandex, 404 and 410 mean the document doesn’t exist.
• Server errors (5xx) and 429. This is how a server reports that it couldn’t or refused to process the request. In response, Google temporarily slows down crawling, and soon removes already-indexed addresses like that from the index. If the errors keep recurring, troubleshoot the server, not the individual page.
• Soft 404. The page says “not found,” but returns a 200 code. In Search Console this is a separate reason — “Soft 404.”

Another Search Console status is “Discovered – currently not indexed”: Google has found the page but hasn’t yet added it to the index. Per the help documentation, this usually means crawling was postponed because it could have overloaded the site.

The page is hard to find: no internal links, not in the sitemap

One of the main ways a search engine learns about pages is links. Yandex builds its list of known pages from internal and external links, the Sitemap file and other data; Google likewise discovers new addresses from links on pages it already knows about and from the Sitemap.

A page that no link inside the site points to can go unnoticed for a long time. For more on how robots find pages through internal links, see a separate article.

A Sitemap file is especially useful for large sites and sites where some pages aren’t linked to the rest. But even it only helps: Google says a Sitemap doesn’t fully guarantee that all pages will be crawled and indexed, and Yandex says it doesn’t guarantee that every URL listed in the file will make it into search. For how the Sitemap file works and what to include in it, see a separate article.

Low-value or very similar pages

A technically accessible page can still fail to make it into search. According to Yandex’s help center, the algorithm may leave a page out of search if it has little chance of being in demand among users: for example, the page has no content visible to the robot, it repeats pages the engine already knows, or it doesn’t answer users’ queries. In Webmaster, such a page gets the status “Low-value or low-demand page.” The algorithm’s decision can change: it will re-check an updated page. Google lists insufficiently high-quality content among the common indexing problems.

What to do about it is a decision for each group of pages. Pages you need are worth reworking for a real query. Near-duplicates should be merged. Utility and technical pages that shouldn’t be in search should be closed off with noindex.

What to do after fixing the problem

Recrawl in Webmaster and URL inspection in Search Console

Yandex. The help center recommends submitting the page for a recrawl after fixing the cause. The “Recrawl pages” tool is in the “Indexing” → “Recrawl pages” section; the number of addresses is limited, and the limit resets daily. The request status “Request processed” doesn’t mean the robot has indexed the page. If the page is blocked from indexing, it won’t get into search.

Google. The URL Inspection tool has a “Request indexing” button. It has a daily quota, and submitting a request doesn’t guarantee the page will appear in the index. Google also warns that submitting more requests doesn’t speed up indexing. If you’ve fixed the cause for a whole group of pages in the indexing report, report it with the “Validate fix” button.

Sitemap and internal links as discovery signals

If a lot of pages have changed, submitting them one by one isn’t practical. For a large number of addresses, Google recommends a Sitemap file instead of individual requests. In Yandex, Sitemap is one way to report new and changed pages, alongside recrawl requests, crawling via the Metrica counter, and the IndexNow protocol.

Internal links work continuously: the robot finds a page through them during a normal crawl of the site.

Checklist after fixing the problem:

1. The page opens normally, returns a 200 code, and doesn’t redirect to another address.
2. The page has no noindex — neither in the robots meta tag nor in the X-Robots-Tag header.
3. The page isn’t blocked in robots.txt.
4. Canonical points to the page itself, if it’s a standalone page.
5. The address is in the Sitemap, and internal links point to the page.
6. The page has been submitted for recrawl in Webmaster and for indexing in Search Console (except for pages with the status “Crawled – currently not indexed”); for a group of pages, “Validate fix” has been clicked.
7. The status has been rechecked after some time in “Pages in search” and in the “Page indexing report.”

How not to lose indexing when you change the site

Before-and-after release checks: what to compare (robots, meta robots, canonical, response codes, sitemap)

Indexing is a basic condition for visibility: search can’t make use of a page that’s inaccessible to search engines. Indexing problems can appear after a release — a new template, a structure change, a move to a different platform. The most reliable approach is to record the state of key pages before the changes and compare it with the state afterward.

When a site’s structure changes, Yandex recommends setting up 301 redirects from the old addresses to the new ones, making sure the new pages return a 200 code and aren’t blocked in robots.txt, updating the Sitemap, and submitting the new pages for recrawl. For a design change, it recommends checking that pages have no noindex and no rel=»canonical» pointing to other addresses.

What to compare before and after	Sign of a problem
robots.txt	New Disallow rules have appeared for sections that should be in search
Meta robots and X-Robots-Tag	noindex has appeared on templates that should be indexed
Canonical	Canonical now points to a different address — the homepage, the old URL, a staging domain
Response codes	Pages that used to return 200 now return 404, 5xx or a redirect; redirect chains have appeared
Sitemap	Important addresses have disappeared from the file, or it now contains addresses with noindex, redirects and errors

After a release, it’s worth watching the “Recent changes” tab in the “Pages in search” report and the trend in the “Page indexing report.” Indexing is one of the areas checked during an SEO audit; for what else gets checked during an audit, see a separate article.

Frequently asked questions

What is site indexing, in simple terms?

A search engine has downloaded the page, parsed its content, and stored information about it in its database — the index. Downloading (crawling) and indexing are different steps: a crawled page doesn’t necessarily make it into the index.

How do I check a page in Yandex?

Enter the address in the “Page check” tool in Webmaster — it will show whether the page is in search. The list of pages and reasons for exclusion are in the “Pages in search” report.

How do I check a page in Google?

Through the “URL Inspection tool” in Search Console: it shows the page’s status in the index and the canonical address Google has chosen. The site: operator isn’t reliable for this.

Why might a page not get indexed?

Possible causes include a block in robots.txt, noindex, canonical pointing to a different address, duplicates, a server response other than 200, or low-value content. The exact reason for each page is shown in Webmaster on the “Excluded pages” tab of the “Pages in search” report and in Search Console’s “Page indexing report.” That said, neither Yandex nor Google includes every page in search.

Can indexing be sped up?

You can notify search engines about pages: through “Recrawl pages” in Webmaster, “Request indexing” in Search Console, and the Sitemap file. Yandex says a recrawl helps speed up indexing of new pages; Google says submitting more requests doesn’t speed up indexing. None of these methods guarantees that a page will make it into the index.

Will every page from the Sitemap make it into the index?

No. A Sitemap helps a search engine find pages, but neither Google nor Yandex guarantees indexing of every address in the file.