News
X-Robots-Tag vs Meta Robots for Public Sites
August 13, 2026
X-Robots-Tag vs meta robots for public sites is a delivery-channel decision: the HTTP header works on HTML and non-HTML, while the robots meta tag only exists on HTML documents. Pick the header for PDFs, images, and CDN-served files; pick the meta tag when the document template already owns SEO. Neither is access control, and RoastMyOpsec does not score missing robots directives as a standalone F.
The practical difference
The robots meta tag lives in HTML and is ignored on a PDF or a bare JSON response. X-Robots-Tag is the same vocabulary (noindex, nofollow, nosnippet, and friends) on the HTTP response, so a CDN can mark a download without wrapping it in a page. Google Search Central documents both as equivalent signals when they apply.
Crawlers that never fetch the body will not see a meta tag. Crawlers that never look at headers will not see X-Robots-Tag. Well-behaved search crawlers look at both. Attackers and random clients do not have to. A noindex header does not replace auth, edge deny, or keeping backups off the origin.
Conflicting signals — meta index plus header noindex, or robots.txt Disallow that blocks the crawler from seeing noindex — are how pages linger in a surprise index. Resolve the conflict at the edge instead of stacking folklore.
How to choose based on what you shipped
Ship X-Robots-Tag on non-HTML assets and on HTML if the edge already injects headers. Use meta robots when marketing CMS templates are the source of truth and you cannot touch response headers. Do not rely on either to hide a path that should return 401/403 or not exist.
| Option | When it wins | Watch-out | Takeaway |
|---|---|---|---|
| X-Robots-Tag | PDFs, images, or CDN files need noindex | Easy to forget on one cache layer | Header works without HTML |
| Meta robots | The HTML template already owns SEO | Does nothing on non-HTML responses | Fine for documents only |
| robots.txt Disallow | You want well-behaved crawlers to skip a prefix | Can prevent crawlers from seeing noindex | Guidance, not a lock |
| Authentication / deny | The URL should not be public at all | Indexing tags are not a substitute | This is the actual control |
When X-Robots-Tag still wins
X-Robots-Tag still wins for attachments, print PDFs, and any response that has no <head>. Meta robots still wins when the CMS cannot set headers but can edit the document. Missing either is normal on a brochure homepage you want indexed.
Common mistakes
The first mistake is putting noindex only in HTML on a PDF that search engines still fetch as a file.
The second mistake is Disallowing a URL in robots.txt so crawlers never see the noindex you added.
The third mistake is treating X-Robots-Tag as a substitute for access control.
What a URL roast can prove
RoastMyOpsec’s free pass covers headers (CSP, HSTS, framing, referrer, nosniff, Permissions-Policy), TLS, cookies, path probes such as .git/HEAD and .env, package.json signatures, and js-secrets. It does not grade X-Robots-Tag or meta robots as a standalone F. Confirm crawl hints in your own crawler tools; use the roast for origin leaks.
Free audit the URL you own
RoastMyOpsec is a defensive public-surface roast: headers, cookies, sensitive paths, and more — no exploit payloads. Start with the free audit, then open the vault if the blurred findings look expensive.
Free audit nowFAQ
- Can meta robots noindex a PDF?
- No. PDFs have no HTML meta tag. Use X-Robots-Tag on the file response, or do not expose the file.
- Is noindex a security control?
- No. It is a hint to cooperating crawlers. Anyone who can request the URL can still read it.
- Does RoastMyOpsec fail missing robots tags?
- No. Crawl directives are not a scored F. The roast still reads other response headers that actually affect browsers.