# llms.txt vs robots.txt vs agents.txt: What Each File Controls

> robots.txt controls whether a crawler may fetch your pages (RFC 9309), llms.txt hands language models a curated markdown map of what is worth reading, and agents.txt is an early, unstandardized proposal for declaring what AI agents may do — three files, three different questions, verified 2026-07-29.

## The one-sentence answer

**robots.txt answers "may you crawl?", llms.txt answers "what should you read?", and agents.txt proposes to answer "what may an agent do here?"** robots.txt (1994, standardized as RFC 9309 in 2022) grants or denies crawl access; llms.txt (2024) is a curated markdown map with no blocking power; agents.txt (2025) is a community draft no standards body has adopted. They do not conflict — a site can ship all three.

## The three files at a glance

| File | The question it answers | Format & semantics | Standard status | Adoption (as of Jul 2026) |
|---|---|---|---|---|
| /robots.txt | May this crawler **fetch** these paths? | Plain-text directives — User-agent, Allow, Disallow, plus Sitemap; advisory access control | Standardized — RFC 9309 (Sep 2022); originated 1994 by Martijn Koster | Universal; GPTBot, ClaudeBot, PerplexityBot document compliance officially |
| /llms.txt | What is **worth reading** here? | Markdown — required H1, blockquote summary, H2 link lists; guidance only, no blocking | Community proposal — Jeremy Howard (Answer.AI), Sep 3, 2024; no governance body; no engine committed to consuming it | ≈10% of ~300,000 studied domains (SE Ranking, 2026); adopters incl. Anthropic, Stripe, Cloudflare, Vercel, Mintlify, Supabase (live-verified 2026-07-29) |
| /agents.txt | What may an **agent do** here? | Structured text declaring identity, permissions, services, endpoints — semantics vary by variant | Unstandardized — ≥4 incompatible community drafts (2025); no IANA well-known registration | Experimental; verifiable deployments are essentially the spec authors' own sites |

## What each file actually does

### robots.txt — crawl access control (RFC 9309)

robots.txt **blocks or allows**. Born as Martijn Koster's 1994 consensus and standardized as RFC 9309 (September 2022, authored by Koster with Google's Illyes, Zeller and Sassman), it tells a crawler which paths it may fetch via User-agent/Allow/Disallow groups, plus a Sitemap pointer. AI-relevant in 2026 via per-crawler opt-outs (Google-Extended, introduced 2023-09-28, controls Gemini training without touching Search — see [the crawler registry](/crawlers)) and Cloudflare's Content Signals Policy (2025-09-24), which [extends robots.txt with usage preferences](/protocols/content-signal). Its structural limit: advisory. GPTBot, ClaudeBot and PerplexityBot document compliance, but nothing enforces it — enforcement is [the access-control dimension](/agent-readiness/access-control).

### llms.txt — a curated map for language models

llms.txt **curates**. Proposed by Jeremy Howard (Answer.AI) on 2024-09-03: a root markdown file — required H1, blockquote summary, H2-sectioned link lists — giving a language model your best content without layout HTML. Adoption is real but partial: SE Ranking's ~300,000-domain study found 10.13% shipping it; Anthropic, Stripe, Cloudflare, Vercel, Mintlify and Supabase all serve one (live-verified 2026-07-29). The honest caveat: **no major engine has committed to consuming llms.txt** — Google's John Mueller publicly compared it to the keywords meta tag (reported April 2025). It curates; it cannot block; it is not a ranking lever. This site ships one anyway — [fetch /llms.txt](/llms.txt) — because it costs minutes and answers an agent's first question: where do I start? See [the how-to](/agent-readiness/llms-txt) and [the Atlas record](/protocols/llms-txt).

### agents.txt — a proposed agent-terms declaration

agents.txt **declares** — or proposes to: a root file stating who the site is, what an agent may do, which services and agentic endpoints exist. Reality as of July 2026: at least four mutually incompatible community drafts (asturwebs reference variant at v2.0 draft, MIT), no standards body, no IANA well-known registration (registry checked — it also lists neither llms.txt nor agents.json). Verifiable deployments ≈ the authors' own sites. A signal of where declaration is heading, not something to build on yet — [the Atlas tracks its variants](/protocols/agents-txt). Disambiguation: [/.well-known/agents.json is the capability manifest this site ships](/agent-readiness/agents-json); AGENTS.md instructs AI **coding** agents in a repository. Three different jobs.

## Can the three files coexist?

Yes — they compose: robots.txt gates access, llms.txt guides reading, agents.txt would declare permissions. A well-equipped 2026 site ships robots.txt with explicit AI policy, llms.txt as the curated map, and — today more usefully than agents.txt — an agents.json manifest. Self-demonstration: this site serves [/robots.txt](/robots.txt) naming the AI crawlers it welcomes, [/llms.txt](/llms.txt), and [/.well-known/agents.json](/agent-readiness/agents-json) — every claim on this page is fetchable from the site making it.

## Which files should your site ship today?

Ship robots.txt with a deliberate AI policy first — the only one with standardized, documented compliance, and [the training-crawler opt-out tokens](/crawlers/training) only work there. Ship llms.txt second: minutes of work, real adopter cohort, useful to agents even without engine commitments — just do not expect rankings. Skip agents.txt until a variant standardizes; declare capabilities in agents.json instead. Whether your files resolve, parse and agree is what the [Agent-Readiness Audit](/services) checks — its discovery and access checks fetch /robots.txt and /llms.txt live.

## llms.txt, robots.txt and agents.txt — frequently asked questions

### Does llms.txt replace robots.txt?

No. llms.txt has no blocking semantics — it is a curated reading map. robots.txt remains the only standardized file that grants or denies crawl access (RFC 9309). They answer different questions and coexist.

### Do AI crawlers actually obey robots.txt?

The major ones document that they do: OpenAI (GPTBot), Anthropic (ClaudeBot) and Perplexity (PerplexityBot) publish official compliance statements. The file is still advisory — a policy reputable crawlers honor, not an enforcement mechanism.

### Is agents.txt a standard I should implement?

Not yet. As of July 2026: at least four incompatible community drafts, no standards-body adoption, no IANA well-known registration. Declare capabilities in /.well-known/agents.json instead and revisit if a variant standardizes.

### What is the difference between agents.txt, agents.json and AGENTS.md?

agents.txt is an unstandardized root-file proposal for agent terms; /.well-known/agents.json is a machine-readable capability manifest (this site ships one); AGENTS.md instructs AI coding agents in a repository. Same naming instinct, three different jobs.

All file-standard facts verified against primary sources on 2026-07-29 — including live fetches of the named adopters' llms.txt files and the IANA well-known-URI registry.

Related: [make your site agent-ready](/agent-readiness) · [discoverability](/agent-readiness/discoverability) · [access control](/agent-readiness/access-control) · [llms.txt in the Atlas](/protocols/llms-txt) · [agents.txt in the Atlas](/protocols/agents-txt) · [the AI crawler registry](/crawlers) · [llms.txt, defined](/glossary/llms-txt) · [audit which files your site ships](/services)