News
AI Crawlers vs Search Crawlers in robots.txt for Public Sites
February 13, 2022
AI crawlers vs search crawlers in robots.txt for public sites is a User-agent grouping decision: keep Googlebot and other search crawlers on the policy you want for search, and add separate groups for named AI crawlers only if you have a real product reason to treat them differently. robots.txt is still not access control. RoastMyOpsec reads Disallow lines for treasure-map shapes; it does not score AI crawler groups, llms.txt, or IndexNow as standalone Fs.
The practical difference
RFC 9309 is the Robots Exclusion Protocol. Well-behaved crawlers pick the most specific User-agent group. Search crawlers exist to index public pages you want found. AI training or retrieval crawlers use different names; vendors publish those names and they change. A blanket Disallow for one name does not bind a crawler that ignores robots.txt.
llms.txt is an optional brochure of URLs you are willing to summarize. It is not a robots group. Splitting search and AI in robots.txt is a content policy. Listing backup or admin-shaped prefixes so “the model does not see them” is still a public map for everyone else.
Google publishes crawler overview docs so you can distinguish Googlebot from other Google user agents. Copy-pasting a viral “block all AI” snippet that also hits your search crawler is how the blog disappears from Google.
How to choose based on what you shipped
Want search: allow your public articles to search crawlers, keep a sitemap of those URLs, keep Disallow boring. Want to discourage some AI crawlers: add named groups without touching the search groups, and do not put secrets in either file. Want AI summaries of docs: a calm llms.txt of public docs, not a second Disallow novel.
| Option | When it wins | Watch-out | Takeaway |
|---|---|---|---|
| Search crawler groups | You want ordinary search indexing | A viral snippet that blocks Googlebot too | Keep search policy separate |
| Named AI crawler groups | You have a reason to treat those agents differently | Names change; non-compliant crawlers ignore you | Policy, not a vault |
| llms.txt | You want to point models at public docs | Copying Disallow paths into the brochure | Optional; not scored |
| Auth / edge deny | The resource should not be fetched by anyone | Relying on User-agent strings | The actual control |
When AI-crawler rules still win
Named AI groups still win when legal or product wants a different crawl policy than search and you will maintain the names. Search groups still win for discoverability. A missing AI group is normal. A Disallow list of sensitive prefixes is the roastable miss either way.
Common mistakes
The first mistake is pasting a block-all-AI robots.txt that also stops your search crawler.
The second mistake is Disallowing sensitive prefixes to “hide them from GPT” in a public file.
The third mistake is treating a missing AI User-agent group as a security finding.
What a URL roast can prove
On a consented scan, RoastMyOpsec fetches /robots.txt and may roast Disallow lines that look like a sensitive-area tour. It does not score AI versus search grouping, llms.txt, IndexNow, GPC, or rel=me as standalone Fs. The free pack still covers headers, TLS, cookies, .git/HEAD, .env, package.json signatures, and js-secrets. User-agent rules are not a lock.
Free audit the URL you own
RoastMyOpsec is a defensive public-surface roast: headers, cookies, sensitive paths, and more — no exploit payloads. Start with the free audit, then open the vault if the blurred findings look expensive.
Free audit nowFAQ
- Does robots.txt stop all AI crawlers?
- Only the well-behaved ones that honor RFC 9309 and match your User-agent groups. It is not access control.
- Should I put admin paths in a GPTBot Disallow?
- No. Do not advertise sensitive prefixes in robots.txt. Deny or do not deploy those URLs.
- Does RoastMyOpsec grade AI crawler rules?
- No. It may comment on juicy Disallow lists. Missing AI groups are not an F.