Robots.txt and Noindex Strategy

Robots.txt and noindex tags get confused constantly, but they do genuinely different jobs — robots.txt controls crawling (whether a crawler visits a URL at all), while noindex controls indexing (whether a page, once crawled, is included in search results). Mixing these up is one of the more common and more damaging technical SEO mistakes.

Robots.txt should block crawling, not hide already-indexed content

A page already indexed, then later blocked via robots.txt, doesn’t get removed from search results — it just becomes a URL Google can no longer re-crawl to update or confirm its status, which can leave an outdated version indexed indefinitely. Robots.txt is the wrong tool for removing something already in search results.

Noindex is the correct tool for “don’t show this in search”

A page that should genuinely never appear in search results — an internal utility page, a thin filtered view, a duplicate variation — should use a noindex meta tag or header, not a robots.txt block. Critically, the page must remain crawlable for the noindex tag to even be seen and respected.

The most damaging mistake: blocking noindexed pages from being crawled

If a page has both a robots.txt block and a noindex tag, the crawler may never actually revisit the page to see the noindex directive, especially if it was indexed before the block was added — this can result in a page staying indexed indefinitely despite the clear intent to remove it.

Staging environments need a real noindex strategy, not an assumption

A staging or preview deploy should be noindexed sitewide, ideally combined with basic access restriction — relying purely on “nobody will find the URL” is not a real strategy, and staging content accidentally getting indexed is a common, avoidable problem.

Review both files together, periodically

Robots.txt rules and noindex tags should be reviewed together as one coherent strategy, not managed independently — a periodic check confirms neither is accidentally undermining the other.

Where this fits

Getting crawl and index control right is foundational and often overlooked until something’s already gone wrong. A Free Strategy Audit includes a real check of both for a specific site.