Robots Meta Tags and Noindex: The Decisions That Matter
Page-level instructions, in a meta tag or X-Robots-Tag header, that control whether a page is indexed and how its snippet can be shown.
Samir Haddad
Why does it matter?
Noindex is the reliable way to keep a crawlable page out of search results, but it only works if search engines are allowed to crawl the page and see it.
What are the numbers?
- noindex tells search engines not to show the page in results
- X-Robots-Tag the HTTP header equivalent, useful for PDFs and other non-HTML files
- robots.txt conflict a page blocked in robots.txt cannot have its noindex seen
- Other directives nofollow, nosnippet, max-snippet and max-image-preview
What should I do?
- Use noindex for pages that should be crawlable but not indexed
- Use X-Robots-Tag for non-HTML files
- Check that noindexed pages are not blocked in robots.txt
- Audit noindex tags after deployments
- Remove noindexed pages from sitemaps
What should I avoid?
Avoid:
- Blocking pages in robots.txt to remove them from search
- Staging noindex tags pushed to production
- Noindex on pages that should rank
- Conflicting canonical and noindex signals
When should I get help?
Short answer Bring in help when important pages disappear from search after a release, or when many pages show as excluded by noindex.
Where this comes from
- Google Search Central — Block search indexing with noindex
- Google Search Central — Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
The figures and practices above come from the sources listed.
Working on something like this?
We take on SEO Services work for teams who want it done once, properly. Tell us what you are building and we will tell you honestly whether we are the right studio for it. Start a project.
Where to go next
Spotted something wrong? Report an error on this page. We correct on the page and say what changed.
