llms-txt-adoption-2026

llms.txt adoption among top sites — 8.8 % of the Tranco top 1,000

as of 2026-09-08 · published 2026-06-17 · author Özden Erdinc

What's new? Updated September 2026. Dates come from this page's git history, never from a global stamp — and they move only when the content moves.

7 Sep 2026: GEO pass after Alpar et al. (Rheinwerk 2026): every page shows its real publication and update dates derived from its git history (never a global stamp), a 'What's new?' box fed by this changelog, and an author box with verifiable profiles; the 32 guide pages got question-form headings with the answer in the first sentence, inline (organization, year) references, a Sources block with verified links and a mini-FAQ mirrored in FAQPage markup; every table carries a caption and header scopes; robots.txt names Googlebot, Applebot, CCBot and Meta-ExternalFetcher in their own groups; license.xml uses the RSL 1.0 usage tokens and media type and is linked from every page with rel="license"; the About page publishes the editorial guidelines; the audit form asks how you found us (counted anonymously in /api/analytics).
10 Aug 2026: Technical SEO audit on all 258 pages: titles trimmed to 60 characters, meta descriptions to 92–155 characters without cut sentences, a BreadcrumbList on every page, og:description kept equal to the meta description.
2 Jul 2026: Full-site audit pass: every Almanac hub now links each record's canonical item page (visible, markdown twin and JSON-LD), GET on the POST-only /api/audit and /api/pay returns a 405 with usage instead of a 404, /data/trusted-keys.json is now served as agents.json promises, /favicon.ico exists, and data corrections landed (robots.txt adherence is now tri-state with 'unclear', Magistral Medium release date, two source stubs restored to honest wording).
17 Jun 2026: First published.

metric: llms.txt adoption among top sites
metric_family: standard-adoption
measures_pillar: /agent-readiness
entity_measured: llms.txt
value: 8.8
unit: % of the Tranco top 1,000
date: 2026-09-06
period: 2026-Q3
figure_kind: cited
source: https://untilnextsession.com/journal/who-serves-llms-txt-and-who-blocks-the-crawlers/
source_or_methodology: Vesper (untilnextsession.com), 2026-09-06: 88 of the Tranco top 1,000 served a valid llms.txt; 121 of the same sample fully blocked at least one named AI crawler. Method, kill rules and per-domain results published with the study. Second, independent measurement on a larger and differently drawn sample: SE Ranking, 7 November 2025, 10.13% of about 300,000 domains (https://seranking.com/blog/llms-txt/).
trend: up
last_verified: 2026-09-08
note: Replaces an earlier '~0' estimate that carried no primary source. The 2026-09-06 figure is a self-published measurement: its author published the method, the kill rules written before the run and the per-domain results, and reported it to this site in the guestbook (/guestbook#gb-4dda9977d945). It has not been independently reproduced. The figure moves with the sample frame, so both measurements are named rather than averaged.

last verified 8 Sep 2026 · by Özden Erdinc

← all State of the Agentic Web · .md · JSON