llms.txt vs robots.txt vs agents.txt: What Each File Controls

robots.txt controls whether a crawler may fetch your pages (RFC 9309), llms.txt hands language models a curated markdown map of what is worth reading, and agents.txt is an early, unstandardized proposal for declaring what AI agents may do — three files, three different questions, verified 2026-07-29.

What do llms.txt, robots.txt and agents.txt each control?

robots.txt answers “may you crawl?”, llms.txt answers “what should you read?”, and agents.txt proposes to answer “what may an agent do here?” robots.txt (1994, standardized as RFC 9309 in 2022) grants or denies crawl access (IETF, 2022); llms.txt (proposed 3 September 2024) is a curated markdown map with no blocking power (Answer.AI, 2024); agents.txt (2025) is an unstandardized draft for declaring agent terms and endpoints.

How do the three files compare at a glance?

Key takeaway: only robots.txt has a standards-track specification and universal adoption; llms.txt has real adopters but no engine commitment and no measurable citation effect (SE Ranking, 2025); agents.txt has neither a converged specification nor a registered name.

How do robots.txt, llms.txt and agents.txt work in practice?

robots.txt — crawl access control (RFC 9309)

robots.txt blocks or allows. Standardized as RFC 9309 (September 2022; authors Koster, Illyes, Zeller and Sassman) from Martijn Koster’s 1994 convention, it tells a crawler which paths it may fetch via User-agent, Allow and Disallow groups, and points to your Sitemap (IETF, 2022). It became AI-relevant through per-crawler opt-outs — Google-Extended, introduced 28 September 2023, is a robots.txt token with no user-agent string of its own that opts your content out of Gemini training and grounding without affecting Google Search inclusion or ranking (Google, 2023; Google, 2026); every AI crawler token is catalogued in the registry — and through Cloudflare’s Content Signals Policy (24 September 2025), which extends robots.txt with usage preferences such as ai-train=no (Cloudflare, 2025). It is advisory: RFC 9309 says its rules “are not a form of access authorization” (IETF, 2022). OpenAI documents robots.txt as the control for GPTBot and OAI-SearchBot, Anthropic states that its bots honor robots.txt directives, and Perplexity points site owners to robots.txt for PerplexityBot; the user-initiated fetchers are the exception — OpenAI says robots.txt rules “may not apply” to ChatGPT-User and Perplexity says Perplexity-User “generally ignores” them (OpenAI, 2026; Anthropic, 2026; Perplexity, 2026). Enforcement is the access-control readiness dimension.

llms.txt — a curated map for language models

llms.txt curates. Proposed by Jeremy Howard of Answer.AI on 3 September 2024, it puts a markdown file at your root — one required H1, an optional blockquote summary, H2-sectioned link lists — so a language model gets your best content without parsing layout HTML (Answer.AI, 2024). Adoption is real but partial: SE Ranking’s study of nearly 300,000 domains, published 7 November 2025, found 10.13% shipping the file and no meaningful correlation with AI citation frequency (SE Ranking, 2025); Anthropic, Stripe, Cloudflare, Vercel, Mintlify and Supabase all serve one (live-fetched 7 September 2026). No major engine has committed to consuming llms.txt: Google’s John Mueller compared it to the keywords meta tag and noted that server logs show AI services do not even request it (reported by Search Engine Journal, 17 April 2025). This site ships one anyway — fetch /llms.txt — because it costs minutes and tells an agent where to start. The llms.txt how-to shows the format; the Atlas records the spec.

agents.txt — a proposed agent-terms declaration

agents.txt declares — or proposes to: a root-level file stating who the site is, what an AI agent may do and which agentic endpoints exist. As of 7 September 2026 there are at least four mutually incompatible drafts — the asturwebs reference variant (v2.0 draft, MIT: markdown with YAML blocks at /agents.txt plus a JSON twin at /api/agents), the dennj variant (key-value directives, MIT), the agents-txt.org capability-declaration spec (directive lines plus an agents.json twin, CC0), and an individual IETF Internet-Draft, draft-srijal-agents-policy-00 (published 7 October 2025, expired 10 April 2026, never adopted by a working group) — while the Atlas records a further variant that renamed itself agent-manifest.txt in early 2026 (Agents Welcome, 2026). No standards body stands behind any of them, and the IANA well-known-URI registry, last updated 19 August 2026, has no entry for agents.txt, llms.txt or agents.json (IANA, 2026). It is a signal of where declaration is heading, not something to build on yet; the Atlas tracks its variants. Do not confuse it with /.well-known/agents.json, the capability manifest this site ships, or with AGENTS.md, the repository-level instruction file for AI coding agents, used by over 60,000 open-source projects (Agentic AI Foundation, 2026).

Can llms.txt, robots.txt and agents.txt coexist?

Yes — they compose: robots.txt gates access, llms.txt guides reading, agents.txt would declare permissions. A well-equipped site in 2026 ships robots.txt with explicit AI-crawler policy, llms.txt as the curated map, and — today more usefully than agents.txt — an agents.json capability manifest. Self-demonstration: this site serves /robots.txt naming the AI crawlers it welcomes, /llms.txt as the curated index, /.well-known/agents.json declaring its actions, and /.well-known/api-catalog, the API-discovery name IANA does register (RFC 9727; IETF, 2025).

Which of llms.txt, robots.txt and agents.txt should your site ship today?

Ship robots.txt with a deliberate AI policy first — it is the only one of the three with a standardized specification and documented crawler compliance, and the training-crawler opt-out tokens only work there. Ship llms.txt second: minutes of work, a real adopter cohort, useful to agents even without engine commitments — just do not expect rankings from it (SE Ranking, 2025). Skip agents.txt until a variant standardizes; declare capabilities in agents.json instead. What a model cannot reconstruct is the verification: the six adopter files, the IANA registry and the four drafts were fetched live on 7 September 2026. Whether the three files you ship actually resolve, parse and agree with each other is what the Agent-Readiness Audit checks — its discovery and access checks fetch /robots.txt and /llms.txt live.

llms.txt, robots.txt and agents.txt — frequently asked questions

Does llms.txt replace robots.txt?

No. llms.txt has no blocking semantics at all — it is a curated reading map. robots.txt remains the only standardized file that grants or denies crawl access (RFC 9309). The two answer different questions and coexist.

Do AI crawlers actually obey robots.txt?

The major crawlers are documented as robots.txt-controlled: OpenAI documents robots.txt as the control for GPTBot and OAI-SearchBot, Anthropic states that its bots honor robots.txt directives, and Perplexity points site owners to robots.txt for PerplexityBot. The exception is user-initiated fetching — OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them — and the file is advisory either way, a policy that reputable crawlers honor, not an enforcement mechanism.

Is agents.txt a standard I should implement?

Not yet. As of September 2026 there are at least four incompatible drafts — one of them an expired individual IETF Internet-Draft — no standards-body adoption and no IANA well-known registration. Declare agent capabilities in a /.well-known/agents.json manifest instead, and revisit agents.txt if a variant standardizes.

What is the difference between agents.txt, agents.json and AGENTS.md?

agents.txt is an unstandardized root-file proposal for agent terms; /.well-known/agents.json is a machine-readable capability manifest (this site ships one); AGENTS.md is a repository file instructing AI coding agents how to build and test a codebase. Same naming instinct, three different jobs.

Primary sources re-verified on 7 September 2026 (first verified 29 July 2026); see the Sources block.

Related: make your site agent-ready · the discoverability dimension · the access-control dimension · llms.txt in the Atlas · agents.txt in the Atlas · the AI crawler registry · llms.txt, defined · audit which files your site ships

← Agent-Readiness · .md