News
llms.txt vs robots.txt on Public Marketing Sites
August 20, 2024
llms.txt vs robots.txt on public marketing sites is two text files that talk to crawlers, not to your WAF. robots.txt (now RFC 9309) tells well-behaved crawlers which paths to skip. llms.txt is a community convention (llmstxt.org) that points language-model crawlers at Markdown or docs you are willing to summarize. Neither authenticates anyone. Disallow is not a vault — it often publishes a treasure map. Putting /admin, /.env, or staging URLs in llms.txt 'so the model understands the product' is the same miss in a new costume. RoastMyOpsec fetches /robots.txt and reads Disallow lines. It does not currently score /llms.txt, does not crawl those URLs, and does not treat a missing llms.txt as an F.
The practical difference
robots.txt is the established crawl policy. sitemap.xml is the index of URLs you want found. llms.txt is an optional brochure for AI systems: links to docs, not secrets. humans.txt is credits. security.txt is researcher contact. Keep the jobs separate.
GEO and answer-engine optimization may want a clean llms.txt that points at public docs. That is a content decision. It is not a reason to weaken robots.txt or to list hidden paths.
How to choose what to publish
You want classic search: keep robots.txt boring and a sitemap of public URLs. You want AI summaries of docs: llms.txt with links you already show in the footer — HTTPS, no tokens in query strings. You do not want either file to name backups, admin, or .env. Auth and edge deny remain the lock. See Disallow vs access control.
| File | Audience | Public origin | Takeaway |
|---|---|---|---|
| /robots.txt | Well-behaved crawlers | Usual; keep Disallow uninteresting | Guidance, not a lock |
| /llms.txt | AI crawlers (community) | Optional; public docs only | Not scored; not access control |
| /sitemap.xml | Index of URLs you want found | Public URLs only | See sitemap vs robots |
| security.txt | Researchers | If you will read Contact | Different job |
When llms.txt still wins
It wins as a short map to public documentation you already publish. robots.txt still wins as the crawl policy. Missing llms.txt is healthy. Chatty Disallow lists are the OPSEC problem either way.
What the roast can prove
Juicy Disallow lines on /robots.txt can surface. llms.txt is not scored. Confirm the files yourself. Pair with sitemap vs robots if a sitemap also listed hidden prefixes.
Common mistakes
The first mistake is copying every Disallow into llms.txt as 'context for the model.'
The second mistake is treating User-agent: GPTBot in robots.txt as a vault.
The third mistake is requiring llms.txt as an OPSEC finding.
Free audit the URL you own
RoastMyOpsec is a defensive public-surface roast: headers, cookies, sensitive paths, and more — no exploit payloads. Start with the free audit, then open the vault if the blurred findings look expensive.
Free audit nowFAQ
- Is a missing llms.txt a vulnerability?
- No. It is an optional content file. Missing it is not a roast F.
- Does llms.txt hide paths from attackers?
- No. Like robots.txt, it is not access control. Do not list secrets or admin URLs in it.
- Does RoastMyOpsec fetch llms.txt?
- Not as a scored path. It does read robots.txt Disallow lines on a consented scan.