News
noindex vs robots.txt Disallow for Public Sites
October 18, 2024
noindex vs robots.txt Disallow for public sites is an indexing decision versus a crawl-budget hint: noindex asks cooperating engines not to index a URL they can still fetch, while Disallow asks them not to crawl a path at all. If a URL must stay reachable for humans, use noindex (and keep it out of the sitemap). If it should not be public, deny it at the edge — Disallow is not a lock and RoastMyOpsec will still roast juicy Disallow lines, not a missing noindex.
The practical difference
Search Central’s block-indexing guidance is blunt: to drop a URL from results while leaving it live, the crawler must be allowed to fetch it and see noindex. Disallowing the path first can leave a stale snippet because the engine never sees the tag. That is a search problem, not a pentest.
robots.txt is a public file. Listing internal, backup, or admin-shaped prefixes as Disallow is how teams publish a tour guide. noindex on a thank-you page or a faceted URL is ordinary SEO. Confusing the two is how staging hosts get a robots.txt novel and a live index entry anyway.
Neither signal binds browsers, scrapers, or anyone who ignores robots.txt. Access control and not deploying the file remain the controls.
How to choose based on what you shipped
Use noindex on public HTML that people need (search result pages, thin filters, thank-you screens) and omit those URLs from the sitemap. Use a short, boring robots.txt for crawler hygiene. Use auth or edge deny for anything that should not be fetched. Do not Disallow a secret and call it OPSEC.
| Option | When it wins | Watch-out | Takeaway |
|---|---|---|---|
| noindex | The URL must load for users but should not rank | Crawler must be allowed to fetch it | Indexing hint, not a lock |
| robots.txt Disallow | You want polite crawlers to skip a prefix | Public file; can block seeing noindex | Keep the list uninteresting |
| Omit from sitemap | You do not want to advertise the URL | Omission is not hiding | Sitemap is an allow-list of wants |
| Edge deny / auth | The resource should not be public | Tags will not save a 200 with secrets | This is the real control |
When robots.txt Disallow still wins
Disallow still wins for crawl budget on infinite calendars or duplicate parameter spaces you already do not want fetched — as long as those paths are not secrets. noindex still wins whenever humans must load the page. A missing noindex on the homepage is not a finding.
Common mistakes
The first mistake is Disallowing a URL you needed noindex on, so crawlers never see the tag.
The second mistake is using Disallow as a treasure map of internal prefixes.
The third mistake is treating either signal as access control.
What a URL roast can prove
On a consented scan, RoastMyOpsec fetches /robots.txt and calls out Disallow lines that look like a sensitive-area tour. The free pack also covers headers, TLS, cookies, .git/HEAD, .env, package.json signatures, and js-secrets. It does not fail you for missing noindex, JSON-LD, Open Graph, RSS, hreflang, GPC, AI crawler groups, or rel=me.
Free audit the URL you own
RoastMyOpsec is a defensive public-surface roast: headers, cookies, sensitive paths, and more — no exploit payloads. Start with the free audit, then open the vault if the blurred findings look expensive.
Free audit nowFAQ
- Will Disallow keep a page out of Google?
- Not reliably. If the URL is already known, blocking crawl can prevent the crawler from seeing noindex. Prefer noindex on live URLs you do not want indexed.
- Is robots.txt a vault?
- No. It is a public hint to well-behaved crawlers. Anyone can read Disallow. Auth and deny are the lock.
- Does the roast score missing noindex?
- No. It may roast chatty Disallow lists. It does not treat ordinary SEO tags as security Fs.