News
rel=describedby vs llms.txt for Public Sites
August 11, 2024
rel=describedby vs llms.txt for public sites is a per-resource description decision versus a site-level AI brochure decision: describedby (an IANA link relation) points a document at a description of that resource, while /llms.txt is a community convention that points language-model crawlers at Markdown you are willing to summarize. Use describedby for accessibility or catalog descriptions of this page; use llms.txt for origin-wide public docs. RoastMyOpsec does not score either as a standalone F.
The practical difference
describedby is a registered link relation: this resource is described by that other resource. In HTML it appears as <link rel="describedby" href="..."> (distinct from aria-describedby, which points at an id in the same document). POWDER and catalog systems used it to attach descriptions without stuffing them into the body.
llms.txt lives at the site root (and optionally /llms-full.txt) and lists URLs of public documentation. RFC 9309 robots.txt remains the crawl policy. Copying Disallow prefixes into llms.txt “for context” is the same treasure-map miss as a chatty robots.txt.
Neither file authenticates. A describedby target that requires a cookie will fail for crawlers and assistive tools that fetch without the session. Keep description URLs public if you advertise them.
How to choose based on what you shipped
Need a machine-readable description of this article or dataset: describedby to a public HTML, RDF, or text description. Want AI crawlers to prefer your docs: a short llms.txt of URLs already in the footer. Want crawl rules: robots.txt, kept boring. Do not use any of them as a vault.
| Option | When it wins | Watch-out | Takeaway |
|---|---|---|---|
| rel=describedby | This resource has a separate public description | Pointing at a cookie-gated or unpublished URL | Per-resource description link |
| llms.txt | You want a site-level list of public docs for models | Listing unpublished or sensitive prefixes | Optional brochure, not a lock |
| robots.txt | You want crawl policy for well-behaved agents | Disallow as a tour guide | RFC 9309 guidance |
| aria-describedby | The description is already in this page’s DOM | Confusing it with a link relation to another URL | Same-document a11y, not llms.txt |
When rel=describedby still wins
describedby still wins for a dataset or article that has a canonical public description on another URL. llms.txt still wins as a one-file map of docs you already publish. Missing both is healthy. Chatty inventories of internal paths are the miss in any of these files.
Common mistakes
The first mistake is pointing describedby at a description URL that is not actually public.
The second mistake is copying robots.txt Disallow paths into llms.txt as model context.
The third mistake is treating a missing llms.txt or describedby link as a security finding.
What a URL roast can prove
RoastMyOpsec does not score rel=describedby, llms.txt, JSON-LD, Open Graph, or AI crawler groups as standalone Fs. It does read robots.txt Disallow lines and still runs the free pack: headers, TLS, cookies, .git/HEAD, .env, package.json signatures, and js-secrets. Keep description links on public documents you already meant to share.
Free audit the URL you own
RoastMyOpsec is a defensive public-surface roast: headers, cookies, sensitive paths, and more — no exploit payloads. Start with the free audit, then open the vault if the blurred findings look expensive.
Free audit nowFAQ
- Is describedby the same as aria-describedby?
- No. aria-describedby references an element id in the same document. rel=describedby is a link to another resource.
- Can llms.txt replace robots.txt?
- No. robots.txt is crawl policy. llms.txt is an optional content map for models.
- Does RoastMyOpsec fetch llms.txt?
- Not as a scored path. Missing llms.txt is not an F. Juicy robots.txt Disallow lines can still surface.