Discoverability — Declaring Your Site to Agents

The agent-readiness dimension that declares your site to AI agents with llms.txt, sitemaps, robots and agents.json so they can find you before reading you.

What does the discoverability dimension mean?

Discoverability is the agent-readiness dimension that lets an AI agent find your site and its key pages before it reads or acts. It is the declaration layer: machine-readable files at fixed paths — /llms.txt, /robots.txt, /sitemap.xml and /.well-known/ resources — that tell an agent what you publish and where. It ships first because an agent cannot read, call or pay a site it cannot find.

Which signals and standards does discoverability cover?

Five declarations, four with a fixed path defined by a public spec:

• llms.txt — a markdown file at /llms.txt whose only required section is an H1, followed by a summary blockquote and H2 link sections; the same spec asks for a clean .md copy of each page (llmstxt.org, 2026). llms-full.txt is its expanded single-file companion.
• robots.txt and the XML sitemap — RFC 9309 (IETF, 2022) puts crawl rules in /robots.txt, matches a crawler to the group naming its product token and falls back to the * group only when no group matches; the Sitemaps protocol caps a file at 50,000 URLs and 50 MB and lets robots.txt point to it (sitemaps.org, 2016).
• .well-known discovery — RFC 8615 (IETF, 2019) reserves the /.well-known/ prefix; RFC 9727 (IETF, 2025) adds /.well-known/api-catalog, a Linkset (application/linkset+json) listing a publisher's APIs.
• agents.json — a manifest at /.well-known/agents.json; Wildcard AI's spec builds it on OpenAPI to describe flows an agent can execute (Wildcard AI, 2025). Its location is discoverability; its actions belong to the capabilities dimension.
• NLWeb — Microsoft's open project, announced in May 2025, that lets users query a site's content in natural language; every NLWeb instance is also an MCP server (Microsoft, 2025).

Two files are often mistaken for site discovery: AGENTS.md is a repository file for coding agents — used by more than 60,000 open-source projects, stewarded by the Agentic AI Foundation (agents.md, 2026) — not a site index, and agents.txt exists only as competing community drafts.

How does the Agent-Readiness Audit score discoverability?

The Audit's Discovery category holds 4 of its 18 checks and 25 of 100 points: llms-txt (8), robots-ai-aware (7), sitemap (6) and llms-full (4) (Agents Welcome, 2026). Every check is a fetch with a pass/fail rule: /llms.txt and /llms-full.txt must return 200 and start with a markdown heading; /robots.txt must name at least one AI crawler such as GPTBot, ClaudeBot or PerplexityBot; /sitemap.xml must return a <urlset> or <sitemapindex>. agents.json is scored under Capability, not Discovery. This site serves all four live — /llms-full.txt is assembled from the live pages on every request — and its self-audit scores 100/100, asserted by the test suite on every run.

Discoverability — frequently asked questions

Does llms.txt replace robots.txt or the sitemap?

No. llms.txt is a curated markdown index for language models, robots.txt (RFC 9309) states crawl permissions, and the sitemap enumerates URLs. An agent-ready site ships all three at their fixed paths.

Where must the discovery files live?

At fixed paths: /llms.txt, /robots.txt and /sitemap.xml at the site root, and manifests such as agents.json and api-catalog under /.well-known/, the prefix RFC 8615 reserves for site-wide metadata.

Is agents.json a discoverability or a capabilities signal?

Both. Its fixed location at /.well-known/agents.json makes it discoverable; the actions it declares are the capabilities dimension, which is where the Agent-Readiness Audit scores it (agents-manifest check, 7 points).

Related: what llms.txt, robots.txt and agents.txt each control · the llms.txt spec · the agents.json spec · implement llms.txt · all six readiness dimensions · audit your site

← Agent-Readiness · .md