Noindex, nofollow and meta robots: what to block from indexing, and how

Noindex, nofollow and meta robots: what’s what

What noindex is

Indexing is the processing of a downloaded page and storing information about it in a search engine’s database. Only an indexed page can appear in search.

noindex blocks a page from being indexed. When Google finds the rule while crawling, it excludes the page from search results, even if other sites link to it. Yandex’s help states it directly: a page with noindex won’t take part in search results.

The phrase «a page is closed to indexing» means exactly this: it’s not in search. But the crawler still downloads it — otherwise it couldn’t read the rule. This matters for everything that follows.

What nofollow is: a page rule and a link attribute

The word nofollow («don’t follow») is used in two different places.

• nofollow in the robots meta tag applies to every link on the page. Google describes it as «don’t follow the links on this page,» Yandex as «don’t follow the links on the page.»
• nofollow in the rel attribute is set on a single link. Google lists it as a way to tell it you don’t want to associate your site with that link. Yandex lists rel="nofollow" among the ways to hide a link from indexing.

Neither of the two values removes from search the page the link itself sits on.

Meta robots and X-Robots-Tag: where the rules are set

The robots meta tag is a line in the <head> section of a page’s HTML with rules for search crawlers.

X-Robots-Tag is an HTTP header that the server sends along with a page or a file. It can carry the same rules as the meta tag. Yandex also supports this header.

Four mechanisms and robots.txt: a comparison

Mechanism	Where it’s set	What it affects	Google	Yandex
The robots meta tag	In the <head> of the HTML page	The whole page	Supported; the set of rules is in the table below	Supported
X-Robots-Tag	An HTTP response header from the server	A page or a file: PDF, image, video	The same rules as the meta tag	Supported
The rel attribute on a link	In the <a> link tag	A single link	The robot usually doesn’t follow the link, but may find the page other ways	Listed among the ways to hide a link from indexing
The <noindex> element	Around a fragment of HTML code	Part of the page’s text	Not documented by Google	Supported
Disallow in robots.txt	A file at the site’s root	Crawling of addresses, not the index	Indexing rules on a blocked page are ignored	Blocked pages can still appear in search

More on the last row in the section «Why blocking crawling doesn’t remove a page from search.»

The robots meta tag and X-Robots-Tag: syntax and directives

How to write the <meta name="robots"> tag

The meta tag is placed in the page’s <head> section. Google also takes into account a robots meta tag placed in <body>. The name attribute specifies which robots the rule is for, and content carries the rule itself. Multiple rules are listed separated by commas. Google also allows a different approach — several separate meta tags. Google reads the name and content attributes case-insensitively.

<!-- Don't index the page and don't follow the links on it -->
<meta name="robots" content="noindex, nofollow">

The value robots in the name attribute addresses the rule to every robot. To set a rule for just one system, the name of its robot is used instead of robots:

<!-- Google only -->
<meta name="googlebot" content="noindex">

<!-- Yandex's robots only -->
<meta name="yandex" content="noindex">

The googlebot version is described in Google’s help, the yandex version in Yandex’s help. Google warns that other search engines may interpret noindex differently.

Google notes that a content management system most likely has a way to generate these meta tags. Either way, it’s worth checking the page’s final code rather than the setting in the admin panel.

X-Robots-Tag: rules in the HTTP header

This header is needed where there’s nowhere to put a meta tag: in PDFs, video, images, and other non-HTML files. It’s also convenient when it’s simpler to set the rule on the server for a whole group of addresses.

X-Robots-Tag: noindex

Google supports the same rules in the header as in the meta tag. For the header, Yandex lists noindex, nofollow, none and noarchive, with multiple values separated by commas: X-Robots-Tag: noindex, nofollow. Google’s help gives an example for all PDF files on a site — in the Apache and NGINX configuration:

# Apache: header for all PDF files
<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

# NGINX: the same thing
location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

Where exactly the configuration is stored depends on the server and hosting. A rule set in the header isn’t visible in the page’s HTML code, so it’s easy to forget about it — checking for it is covered below.

Which directives Google and Yandex support

The table lists directives from both systems’ help as of the check date. «Not in the help» means the system doesn’t document the directive, not that it «definitely doesn’t work.»

Directive	What it does	Google	Yandex
noindex	Don’t show the page in search results	Yes	Yes
nofollow	Don’t follow the links on the page	Yes	Yes
none	Same as noindex, nofollow	Yes	Yes
all	No restrictions	Yes; this is the default value	Yes; equivalent to index and follow
index, follow, archive	Permissive directives	Not in the help	Override restrictive directives
noarchive	Don’t show a link to a cached copy	No longer used: Google doesn’t have a cached-copy link	Yes
nosnippet	Don’t show a text snippet or video in the results	Yes	Not in the help
max-snippet, max-image-preview, max-video-preview	Limit the length of the snippet, the size of the image, and the length of the video in results	Yes	Not in the help
notranslate	Don’t offer a translation of the page in results	Yes	Not in the help
noimageindex	Don’t index images on the page	Yes	Not in the help
unavailable_after	Don’t show the page after a specified date and time	Yes	Not in the help
indexifembedded	Allow indexing a page’s content when it’s embedded in another page	Yes	Not in the help

Display directives like nosnippet control the snippet shown in search results. A separate article covers how such restrictions relate to AI services’ answers: how display directives affect AI answers.

What happens when directives contradict each other

Here Google and Yandex behave differently.

Google applies the strictest of the conflicting rules. An example from its help: if both max-snippet:50 and nosnippet apply to a page, nosnippet wins.

Yandex gives priority to permissive directives. An example from Yandex’s help:

<meta name="robots" content="all">
<meta name="robots" content="noindex, follow">
<!-- Yandex will choose all: the text and links will be indexed -->

The practical takeaway: if a page ends up with two meta tags — say, one from the theme and one from an SEO plugin — or a meta tag and a header with different rules, the outcome can differ between the two systems. Say one meta tag has noindex and the other has all. By the rules in each help center, such a page would be closed in Google and open in Yandex. So check the page’s final response, not a single setting.

«noindex, follow»: what it means

This combination means: don’t index the page, but there’s no restriction on following its links.

• Google: follow isn’t in the list of rules in its help; by default there are no restrictions. Without nofollow, Google’s robots can use the links on a page to find other pages. So for Google, noindex, follow is essentially the same as noindex: the rule doesn’t restrict the links.
• Yandex: follow is a permissive directive, and the robot uses permissive directives by default.

There’s no need to specify follow explicitly. Neither help center describes what the search engines do with the links on a page that stays out of the index for a long time. So don’t build important internal linking on such pages.

Nofollow for links: rel="nofollow", sponsored and ugc

How to set nofollow on a link

The value is set in the rel attribute of the link tag. Google has three values:

• sponsored — for ads and paid placements;
• ugc — for links published by users: in comments, on a forum;
• nofollow — when neither of the other two values fits, and you don’t want Google’s robots to follow the link and associate it with your site.

<a href="https://example.com/" rel="nofollow">Link text</a>
<a href="https://example.com/" rel="sponsored">Link text</a>
<a href="https://example.com/" rel="ugc nofollow">Link text</a>

Values can be combined, separating them with a space or a comma: rel="ugc nofollow" or rel="ugc,nofollow". Yandex’s help shows the same kind of notation: rel="nofollow sponsored" and rel="nofollow,sponsored".

How Google and Yandex treat rel

Google writes that its robots usually don’t follow links with these rel attributes. But a robot can still find and crawl the target page other ways — for example, through a sitemap file or links from other sites. So rel="nofollow" doesn’t hide a page from Google.

Yandex lists rel="ugc" (forums and reviews), rel="sponsored" (ad links) and rel="nofollow" (so the robot doesn’t follow the link) in its section on how to hide a link from indexing. Its help doesn’t specify whether the robot always follows this instruction.

One link, or every link on the page

If you need to block one link, set rel on it. If no link on the page should be followed at all, nofollow in the robots meta tag will do. The meta tag then applies to every link without exception, including the menu and internal links.

For regular links that the robot is fine to follow, the rel attribute isn’t needed. On internal links, nofollow usually doesn’t solve the «remove the page from search» task: the robot can still find it through other links or the sitemap. For that you need noindex on the page itself, left open for crawling. We have a separate article on how to build internal links between pages.

Yandex’s <noindex> tag: how to hide part of the text

The <noindex> element hides a fragment of text from Yandex’s indexing, not the whole page. It’s placed around the relevant piece of HTML code. If code validity matters, Yandex’s help offers a version using HTML comments instead.

<noindex>text that shouldn't be indexed</noindex>

<!--noindex-->text that shouldn't be indexed<!--/noindex-->

According to Yandex’s help, the element isn’t sensitive to nesting and can be placed anywhere in the HTML code. If the closing tag is missing, the entire content of the page is considered hidden. So every opened <noindex> needs to be closed.

Google’s documentation doesn’t describe the <noindex> element. It isn’t in the list of meta tags and attributes that Google’s robots recognize. Don’t count on it to hide text from Google.

<noindex> around a link vs. rel="nofollow": what’s the difference

These are different mechanisms, even though they’re often used together.

• <noindex> hides a fragment’s text from being indexed by Yandex.
• rel="nofollow" applies to the link itself: according to Yandex’s help, so the robot doesn’t follow it; Google usually doesn’t follow such a link either.
• A meta tag with noindex removes the entire page from search.

What Yandex does with a link inside <noindex> isn’t separately described in its help. If the goal is to block the link, the rel attribute is there for that.

How to block a page, a section, or an entire site from indexing

Task	Method	Why not something else
A utility page	noindex in the meta tag or in X-Robots-Tag	Disallow doesn’t remove the page from search
A section or group of pages	noindex in the section template, or X-Robots-Tag on the server	Setting the rule by hand on every page is unreliable: new pages in the section will be left without it
PDFs and other files	X-Robots-Tag	A meta tag can’t be placed in a non-HTML file
The entire site, a subdomain, a staging copy	Password or login access, noindex on every page	A password blocks the site from both people and robots; robots.txt only applies to its own host
A page already in search	noindex with crawling left open, 404 / 410 response codes, removal tools	Blocking crawling would stop the robot from seeing the noindex
Duplicates and variants of one page	Not noindex, but canonical — a separate article covers how to choose between canonical, a 301 redirect, and noindex	noindex removes the page from search but doesn’t specify the primary version
A single link	The rel attribute on the link	noindex and nofollow in the meta tag apply to the entire page

A single page

A robots meta tag with noindex, or the X-Robots-Tag: noindex header, both work. This is usually how utility addresses get blocked: on-site search results, the cart, checkout, the account area. The article on site structure covers which utility pages are usually unnecessary in search.

Don’t include a page with noindex in your sitemap. Google advises adding to the sitemap the addresses you want to see in search results. A separate article covers which pages not to include in a sitemap in more detail.

A section or group of pages

For a section, it’s more convenient to set the rule once: in the section’s page template, or via the X-Robots-Tag header on the server for every address matching a path pattern. That way new pages in the section get the rule automatically.

Blocking crawling of a section in robots.txt saves the robot’s requests, but doesn’t remove the pages from search. Why — in the section below.

The entire site, a subdomain, or a staging copy

Password access works well for a staging copy. Yandex names login authentication on a site as one way to block indexing. For pages that shouldn’t appear in search, Google advises noindex or password-only access. A password also keeps outsiders off the site, not just robots.

If a password isn’t an option, you need noindex on every page — via a meta tag in the shared template, or a header on the server.

The rules apply to their own host. robots.txt only covers the protocol, host and port where it’s placed: a subdomain like https://test.example.com/ needs its own file. The meta tag and the header are also set separately for each response: noindex on the main domain doesn’t block the subdomain.

Some CMS platforms have a setting that asks search engines not to index the site. It doesn’t replace checking the pages’ final response. And the reverse case: when moving a staging copy to the live site, check that the block didn’t travel over along with the template.

A page already in search: how to remove it

Google. Add noindex and leave the page open for crawling. Then, in Search Console’s URL Inspection tool, request a re-crawl. The changes show up in the «Page indexing report.»

The Removals tool in Search Console lets a site owner temporarily exclude pages from search results. To remove content permanently, the help requires additional steps: delete the page with a 404 or 410 response, add password protection, or add noindex. Blocking through the tool doesn’t stop Google from crawling the page.

Yandex. Its help first suggests blocking indexing with one of three methods: Disallow in robots.txt, a 404, 403 or 410 response, or a robots meta tag with noindex. Removal can then be sped up with the «Remove pages from search» tool in Webmaster’s Tools section — for individual addresses or by prefix, meaning a whole group of addresses at once. For prefix removal, the help sets a condition: the robot will only remove such pages if Disallow is set for them in robots.txt.

The article on site indexing covers how to resubmit a page for crawling and read indexing statuses.

Why blocking crawling doesn’t remove a page from search

Disallow and noindex: why they don’t work together

The robot’s order of operations is this. First it checks whether crawling the address is allowed in robots.txt. Only then does it download the page’s code and read the rules in the meta tag and the header. If crawling is blocked, the robot never gets to the second step.

Google states this directly: if crawling a page is blocked in robots.txt, its indexing and display rules will be ignored. The robot won’t process the code and won’t discover the noindex. The page’s content may still appear in results — for example, if other resources link to it.

Yandex’s help on meta tags states: if a page is blocked in robots.txt, the meta tag or header directive doesn’t take effect. Its help on robots.txt says that pages restricted there can still appear in search, and recommends noindex in the code or the header for removing a page from search.

At the same time, Yandex’s help on blocking indexing lists Disallow as one of the methods, and the Webmaster tool requires Disallow for removing a group of addresses by prefix. For a single page, pick one mechanism. If you need the robot to read the noindex, the page must be open for crawling. The article on how to write Disallow rules in robots.txt covers how to write the rules.

When blocking crawling in robots.txt is still the right call

Google describes robots.txt primarily as a tool for managing the crawl traffic robots generate on a site. Blocking crawling makes sense for addresses the robot has no reason to download and that aren’t in search. Examples include endless sorting variants or technical addresses that neither people nor search need.

If such addresses have already made it into search, the rules above imply this order:

1. Open the pages up for crawling and add noindex.
2. Wait until the pages drop out of search — you can see this in Search Console’s and Webmaster’s reports.
3. If crawling these addresses still isn’t needed, add Disallow.

If you start with the third step, the robot won’t see the noindex, and the pages may stay in search.

How to check whether a page is blocked from indexing

The page’s code and HTTP headers

Open the page’s source code in your browser and look for name="robots", as well as name="googlebot" and name="yandex". Pay attention to the number of meta tags: two rules with different values are a reason to figure out where the second one came from.

Response headers are visible in the browser’s developer tools, on the network requests tab. From the command line, you can request them like this:

curl -I https://example.com/page/

Look for the X-Robots-Tag line in the response. Check the page’s final response, not the settings in the admin panel: the rule could have been added by the theme, a plugin, or the server.

Google Search Console

The URL Inspection tool has «Crawl allowed?» and «Indexing allowed?» fields. The first shows whether robots.txt is blocking crawling, the second whether noindex is blocking indexing. The «Test live URL» button checks the page’s current version, and «Request indexing» sends a crawl request.

In the «Page indexing report,» pages with noindex fall into the group with the reason «Excluded by ‘noindex’ tag.»

Yandex Webmaster

In the «Indexing → Pages in search» report, on the excluded pages tab, pages with noindex get the status «Blocked by the noindex element.» The «Indexing → Page check» tool shows the response code, the page’s content as the robot sees it, and its status in search.

If a page isn’t in search for a different reason, work through the diagnostic tree of reasons a page isn’t indexed, step by step.

Post-release checklist

• Page templates that should be in search have no noindex, neither in the meta tag nor in X-Robots-Tag.
• The staging copy is blocked by a password or noindex, and the live site isn’t.
• Pages meant to leave search via noindex aren’t blocked in robots.txt.
• Pages with noindex aren’t in the sitemap.
• The page doesn’t have two robots meta tags with conflicting rules.
• Every <noindex> in the code is closed.
• rel="nofollow" is only set on the links where it’s intended.

Common mistakes

Mistake	The risk	How to fix it
noindex was left in the template after moving from staging, or after a release	Pages that should be in search get excluded from the results	Check the meta tag and headers on key templates after every release
noindex and Disallow on one page	The robot won’t read the noindex, and the page may stay in search	Remove Disallow until the page drops out of search
Pages with noindex in the sitemap	A sitemap lists the addresses you want to see in search — the signals contradict each other	Remove such addresses from the sitemap
noindex instead of canonical for duplicates	The duplicate disappears from search, but the primary version isn’t specified	Use canonical for duplicates (see the table above)
Two conflicting meta tags, or a meta tag and a header with different rules	Google will apply the stricter rule, Yandex the permissive one	Keep one rule in one place
An unclosed <noindex>	For Yandex, the entire page’s content is considered hidden	Close the element, or use the HTML-comment version
Relying on <noindex> or rel as a block for Google	<noindex> isn’t documented by Google; Google usually doesn’t follow a link with rel, but may find the page another way	To remove a page from search, use noindex on the page itself
nofollow in the meta tag on every page instead of rel on specific links	The block applies to every link on the page, including the menu	Set rel on the specific links instead
Rule set in X-Robots-Tag on the server and forgotten	It’s invisible in the HTML code, but the rule is still in effect	Check the response headers, not just the page’s code

If blocked pages turn up after a release, or the template, a plugin, and the server all set rules at once, it’s worth investigating across the whole site, not page by page. This is a technical SEO task — more on how a meta robots and crawling-rules review works as part of technical SEO.

Frequently asked questions

What does «URL blocked from indexing by the noindex tag» mean, and what should I do about it?

This usually describes a situation where a page has noindex on it. In Yandex Webmaster, such a page gets the status «Blocked by the noindex element»; in Search Console, the reason is «Excluded by ‘noindex’ tag.» Check the meta tag, the X-Robots-Tag header, and the page template. If the page should be in search, remove the rule and resubmit it for crawling.

Which file blocks search robots from indexing pages?

There’s no such file. robots.txt controls crawling: Google writes that it isn’t meant for blocking display in search, and Yandex says that pages blocked in it can still appear in search. Blocking indexing is done with the robots meta tag or the X-Robots-Tag header.

Is it true that a nofollow link doesn’t pass along «weight»?

Neither Google’s nor Yandex’s help on the rel attribute uses the concept of «weight.» Google describes nofollow this way: you don’t want robots to follow the link and associate it with your site; the target page may still be found by other means. Yandex: «so the robot doesn’t follow the link.» Neither help center describes how this affects ranking.

Should nofollow be added to every outbound link?

No. Google asks that ad and paid links be marked as sponsored, user-generated ones as ugc, and nofollow used when neither value fits and you don’t want to associate your site with the link. Regular links don’t need the rel attribute.

Is there a «dofollow» attribute?

No. There’s no such value in the lists of rel values in Google’s or Yandex’s help. A regular link with no rel attribute is already a link the robot is free to follow.