What is noindex?

noindex is a directive asking that a page not be added to a search engine’s index, given either with the robots meta tag inside the page or with the server’s X-Robots-Tag response header.

Google does not index a page on which it sees this directive; if the page is already in the index, Google removes it from search results at the next crawl. The directive can be written in two places. On HTML pages, the robots meta tag in the head of the page is given the value “noindex”; to target only Google, the tag’s name can also be written as googlebot. For non-HTML files such as PDFs or images, the same directive is given in the server’s response header, with X-Robots-Tag.

The most important rule is that noindex and robots.txt can work against each other. Google writes that for noindex to work, the page must not be blocked by robots.txt: if the crawler cannot access the page, it cannot read the directive inside it either, and the page may keep appearing in results if it receives links from other sites. The right tool for removing a page from the index is noindex; robots.txt stops crawling, it does not remove anything from the index.

When is noindex used?

• Pages that should never reach a search result: thank-you pages after a form, internal site search results.
• Files that repeat the content of another page.
• Error pages: there is no point in a 404 page itself being indexed.

The most expensive mistake runs the other way: a noindex placed on the whole site during development that someone forgets to remove at launch. The site works, visitors notice nothing, but Google drops the pages from the index one by one. That is why searching the source code for noindex is one of the first checks when the complaint is “my site doesn’t show up on Google”; you will find the nine-point list that includes this check in the guide to not showing up on Google. If you would like to do the check with a specialist, SEO consulting starts with this scan.

First-hand example: on this site, noindex sits in two places, both of them deliberate. The 404 page carries noindex in its own robots meta tag; nobody benefits from an error page appearing in search results. llms-full.txt, which gathers all of the site’s summaries in a single file for AI agents, is not HTML, so it receives the directive in the server’s response header, with X-Robots-Tag: because the file carries a second copy of text that already sits elsewhere on the site, indexing it would mean duplicate content. The same file was not blocked in robots.txt; if it had been, Google could never read the directive. We explained why this file exists in the llms.txt and robots.txt guide.

Whether a page really stays out of the index is tested with the URL Inspection tool in Search Console; the tool also shows whether indexing is allowed. We explained how to read the reasons pages stay out of the index in the “crawled, not indexed” guide.

If you want to find out together why your site doesn’t show up on Google, SEO consulting audits every layer from crawling to indexing and makes the fix in the source code.