News
Last-Modified vs Sitemap lastmod for Public Sites
August 3, 2024
Last-Modified vs sitemap lastmod for public sites is a cache-validator decision versus a crawler-hint decision: Last-Modified (and ETag) help clients skip a body they already have, while <lastmod> in sitemap.xml tells cooperating crawlers when a listed URL changed. Keep both dates honest. Neither is access control, and RoastMyOpsec does not score sitemap lastmod or Last-Modified as a standalone F.
The practical difference
Last-Modified is an HTTP response header. With If-Modified-Since, a client may get 304. It is a caching and bandwidth tool. Sitemap lastmod is XML inside a sitemap the crawler already decided to fetch. Google has said lastmod is useful when it matches reality and ignored when it is stamped “now” on every URL.
A sitemap is an advertisement of URLs you want found. Putting lastmod on a path that should not be public does not hide it; it invites recrawl. Last-Modified on a 200 for .env is still a 200 for .env. Fix exposure, do not timestamp it.
Personalized pages should not be publicly cacheable just because Last-Modified looks old. Cache-Control owns that story.
How to choose based on what you shipped
Set Last-Modified or ETag on static public assets if your origin or CDN can do so cheaply. Set sitemap lastmod only when the URL’s meaningful content changed. Omit lastmod rather than lie. Do not list non-public URLs in the sitemap to “keep lastmod in sync.”
| Option | When it wins | Watch-out | Takeaway |
|---|---|---|---|
| Last-Modified / ETag | Static public files and cache revalidation | Personalized HTML marked public | HTTP caching, not hiding |
| Sitemap lastmod | The listed public URL actually changed | Now-stamping every row | Honesty or omit |
| IndexNow / recrawl APIs | You need to notify supporting engines | Notifying URLs that should not be public | Notification, not a lock |
| robots.txt Disallow | Polite crawlers should skip a prefix | Treasure-map Disallow lists | Guidance, not lastmod |
When Last-Modified still wins
Last-Modified still wins for CDN-cached static assets and simple documents. Sitemap lastmod still wins as a recrawl hint when it is true. Missing either is fine. Lying lastmod and exposing sensitive paths are the misses, and only the latter is roast material.
Common mistakes
The first mistake is stamping sitemap lastmod to now on every URL so crawlers learn to ignore you.
The second mistake is putting lastmod on URLs that should not be in a public sitemap at all.
The third mistake is treating a missing Last-Modified header as a security finding.
What a URL roast can prove
RoastMyOpsec may roast a robots.txt that advertises sensitive prefixes, and it probes a short list of sensitive paths. It does not score Last-Modified, sitemap lastmod, IndexNow, JSON-LD, or hreflang as standalone Fs. The free pack is headers, TLS, cookies, those path signatures, and js-secrets. Confirm sitemaps in Search Console.
Free audit the URL you own
RoastMyOpsec is a defensive public-surface roast: headers, cookies, sensitive paths, and more — no exploit payloads. Start with the free audit, then open the vault if the blurred findings look expensive.
Free audit nowFAQ
- Does Google require lastmod?
- No. It is optional and only helpful when it matches real changes. Fake dates are worse than omitting it.
- Does Last-Modified hide a file?
- No. It is a validator for a response you already served.
- Does RoastMyOpsec grade sitemaps?
- Not as a lastmod QA. Chatty robots.txt Disallow lines can still surface. Hidden-path sitemaps are a separate OPSEC topic.