llms.txt vs robots.txt vs agents.txt: What Each File Controls

robots.txt controls whether a crawler may fetch your pages (RFC 9309), llms.txt hands language models a curated markdown map of what is worth reading, and agents.txt is an early, unstandardized proposal for declaring what AI agents may do — three files, three different questions, verified 2026-07-29.

The one-sentence answer

robots.txt answers “may you crawl?”, llms.txt answers “what should you read?”, and agents.txt proposes to answer “what may an agent do here?” robots.txt (1994, standardized as RFC 9309 in 2022) grants or denies crawl access; llms.txt (2024) is a curated markdown map with no blocking power at all; agents.txt (2025) is a community draft for declaring agent terms and endpoints that no standards body has adopted. They do not conflict — a site can ship all three, and they answer orthogonal questions.

The three files at a glance

What each file actually does

robots.txt — crawl access control (RFC 9309)

robots.txt blocks or allows. Born as Martijn Koster’s 1994 consensus and standardized 28 years later as RFC 9309 (September 2022, authored by Koster with Google’s Illyes, Zeller and Sassman), it tells a crawler which paths it may fetch via User-agent, Allow and Disallow groups, and points to your Sitemap. Two things make it AI-relevant in 2026: per-crawler opt-outs (Google-Extended, introduced September 28, 2023, controls Gemini training without touching Search; every AI crawler token is catalogued in the registry) and Cloudflare’s Content Signals Policy (September 24, 2025), which extends robots.txt with usage preferences like ai-train=no. Its limit is honest and structural: it is advisory. GPTBot, ClaudeBot and PerplexityBot all document compliance officially, but nothing in the file enforces it — enforcement is the access-control readiness dimension.

llms.txt — a curated map for language models

llms.txt curates. Proposed by Jeremy Howard of Answer.AI on September 3, 2024, it puts a markdown file at your root — one required H1, a blockquote summary, H2-sectioned link lists — so a language model gets your best content without parsing layout HTML. Adoption is real but partial: an SE Ranking study of roughly 300,000 domains found 10.13% shipping the file, and Anthropic, Stripe, Cloudflare, Vercel, Mintlify and Supabase all serve one (each live-verified for this page on July 29, 2026). The honest caveat belongs next to the pitch: no major engine has committed to consuming llms.txt — Google’s John Mueller publicly compared it to the keywords meta tag (reported April 2025). It curates; it cannot block, and it is not a ranking lever. This site ships one anyway — fetch /llms.txt — because the file costs minutes and answers an agent’s first question: where do I start? The llms.txt how-to shows the format; the Atlas records the spec.

agents.txt — a proposed agent-terms declaration

agents.txt declares — or proposes to. The idea: a root-level file stating who the site is, what an AI agent is permitted to do, which services and agentic endpoints exist. The reality as of July 2026: at least four mutually incompatible community drafts (the asturwebs reference variant sits at v2.0 draft under MIT), no standards body behind any of them, and no entry in the IANA well-known-URI registry — we checked the registry itself, which also lists neither llms.txt nor agents.json. Verifiable deployments are essentially the authors’ own sites. Treat it as a signal of where declaration is heading, not something to build on yet; the Atlas tracks its variants. Do not confuse it with two similarly named things: /.well-known/agents.json, the capability manifest this site ships, and AGENTS.md, the repository-level instruction file for AI coding agents — same naming instinct, entirely different jobs.

Can the three files coexist?

Yes — they compose, because each answers a question the others cannot: robots.txt gates access, llms.txt guides reading, agents.txt would declare permissions. A well-equipped site in 2026 ships robots.txt with explicit AI-crawler policy, llms.txt as the curated map, and — today more usefully than agents.txt — an agents.json capability manifest. Self-demonstration: this site serves /robots.txt naming the AI crawlers it welcomes, /llms.txt as the curated index, and /.well-known/agents.json declaring its actions — every claim on this page is fetchable from the site making it.

Which files should your site ship today?

Ship robots.txt with a deliberate AI policy first — it is the only one of the three with standardized, documented compliance, and the training-crawler opt-out tokens only work there. Ship llms.txt second: minutes of work, real adopter cohort, useful to agents even without engine commitments — just do not expect rankings from it. Skip agents.txt until a variant standardizes; declare capabilities in agents.json instead. Whether the three files you ship actually resolve, parse and agree with each other is exactly what the Agent-Readiness Audit checks — its discovery and access checks fetch /robots.txt and /llms.txt live.

llms.txt, robots.txt and agents.txt — frequently asked questions

Does llms.txt replace robots.txt?

No. llms.txt has no blocking semantics at all — it is a curated reading map. robots.txt remains the only standardized file that grants or denies crawl access (RFC 9309). The two answer different questions and coexist.

Do AI crawlers actually obey robots.txt?

The major ones document that they do: OpenAI (GPTBot), Anthropic (ClaudeBot) and Perplexity (PerplexityBot) all publish official robots.txt compliance statements. The file is still advisory — it is a policy the reputable crawlers honor, not an enforcement mechanism.

Is agents.txt a standard I should implement?

Not yet. As of July 2026 there are at least four incompatible community drafts, no standards-body adoption and no IANA well-known registration. Declare agent capabilities in a /.well-known/agents.json manifest instead, and revisit agents.txt if a variant standardizes.

What is the difference between agents.txt, agents.json and AGENTS.md?

agents.txt is an unstandardized root-file proposal for agent terms; /.well-known/agents.json is a machine-readable capability manifest (this site ships one); AGENTS.md is a repository file instructing AI coding agents how to build and test a codebase. Same naming instinct, three different jobs.

All file-standard facts on this page were verified against their primary sources on July 29, 2026 — including live fetches of the named adopters’ llms.txt files and the IANA well-known-URI registry.

Related: make your site agent-ready · the discoverability dimension · the access-control dimension · llms.txt in the Atlas · agents.txt in the Atlas · the AI crawler registry · llms.txt, defined · audit which files your site ships

← Agent-Readiness · .md