# Discoverability — Declaring Your Site to Agents

Published 2026-06-17 · Updated 2026-11-08 · Author: Özden Erdinc (https://erdinc.ai) · Editorial guidelines: /about#editorial-guidelines

> The agent-readiness dimension that declares your site to AI agents with llms.txt, sitemaps, robots and agents.json so they can find you before reading you.

## What does the discoverability dimension mean?

Discoverability is the agent-readiness dimension that lets an AI agent find your site and its key pages before it reads or acts. It is the declaration layer: machine-readable files at fixed paths — `/llms.txt`, `/robots.txt`, `/sitemap.xml` and `/.well-known/` resources — that tell an agent what you publish and where. It ships first because an agent cannot read, call or pay a site it cannot find.

## Which signals and standards does discoverability cover?

Five declarations, four with a fixed path defined by a public spec:

- **llms.txt** — a markdown file at `/llms.txt` whose only required section is an H1, followed by a summary blockquote and H2 link sections; the same spec asks for a clean `.md` copy of each page (llmstxt.org, 2026). `llms-full.txt` is its expanded single-file companion.
- **robots.txt and the XML sitemap** — RFC 9309 (IETF, 2022) puts crawl rules in `/robots.txt`, matches a crawler to the group naming its product token and falls back to the `*` group only when no group matches; the Sitemaps protocol caps a file at 50,000 URLs and 50 MB and lets robots.txt point to it (sitemaps.org, 2016).
- **.well-known discovery** — RFC 8615 (IETF, 2019) reserves the `/.well-known/` prefix; RFC 9727 (IETF, 2025) adds `/.well-known/api-catalog`, a Linkset (`application/linkset+json`) listing a publisher's APIs.
- **agents.json** — a manifest at `/.well-known/agents.json`; Wildcard AI's spec builds it on OpenAPI to describe flows an agent can execute (Wildcard AI, 2025). Its location is discoverability; its actions belong to the capabilities dimension.
- **NLWeb** — Microsoft's open project, announced in May 2025, that lets users query a site's content in natural language; every NLWeb instance is also an MCP server (Microsoft, 2025).

Two files are often mistaken for site discovery: `AGENTS.md` is a repository file for coding agents — used by more than 60,000 open-source projects, stewarded by the Agentic AI Foundation (agents.md, 2026) — not a site index, and `agents.txt` exists only as competing community drafts.

## How does the Agent-Readiness Audit score discoverability?

The Audit's Discovery category holds 4 of its 18 checks and 25 of 100 points: `llms-txt` (8), `robots-ai-aware` (7), `sitemap` (6) and `llms-full` (4) (Agents Welcome, 2026). Every check is a fetch with a pass/fail rule: `/llms.txt` and `/llms-full.txt` must return 200 and start with a markdown heading, after YAML front matter if there is any; `/robots.txt` must name at least one AI crawler such as GPTBot, ClaudeBot or PerplexityBot; `/sitemap.xml`, or one of the first three sitemaps named in `robots.txt`, must return a `<urlset>` or `<sitemapindex>`, plain or gzip. `agents.json` is scored under Capability, not Discovery. This site serves all four live — `/llms-full.txt` is assembled from the live pages on every request — and its self-audit scores 100/100, asserted by the test suite on every run.

## Discoverability — frequently asked questions

### Does llms.txt replace robots.txt or the sitemap?

No. llms.txt is a curated markdown index for language models, robots.txt (RFC 9309) states crawl permissions, and the sitemap enumerates URLs. An agent-ready site ships all three at their fixed paths.

### Where must the discovery files live?

At fixed paths: /llms.txt, /robots.txt and /sitemap.xml at the site root, and manifests such as agents.json and api-catalog under /.well-known/, the prefix RFC 8615 reserves for site-wide metadata.

### Is agents.json a discoverability or a capabilities signal?

Both. Its fixed location at /.well-known/agents.json makes it discoverable; the actions it declares are the capabilities dimension, which is where the Agent-Readiness Audit scores it (agents-manifest check, 7 points).

## Sources

1. llmstxt.org: The /llms.txt file — format specification, 2026. https://llmstxt.org/
2. IETF: RFC 9309, Robots Exclusion Protocol, 2022. https://www.rfc-editor.org/rfc/rfc9309.html
3. sitemaps.org: Sitemaps XML format, protocol 0.9, 2016. https://www.sitemaps.org/protocol.html
4. IETF: RFC 8615, Well-Known Uniform Resource Identifiers (URIs), 2019. https://www.rfc-editor.org/rfc/rfc8615.html
5. IETF: RFC 9727, api-catalog: A Well-Known URI and Link Relation to Help Discovery of APIs, 2025. https://www.rfc-editor.org/rfc/rfc9727.html
6. Wildcard AI: agents.json specification, 2025. https://github.com/wild-card-ai/agents-json
7. Microsoft: Introducing NLWeb: Bringing conversational interfaces directly to the web, 2025. https://news.microsoft.com/source/features/company-news/introducing-nlweb-bringing-conversational-interfaces-directly-to-the-web/
8. Agentic AI Foundation: AGENTS.md, 2026. https://agents.md/

Related: [what llms.txt, robots.txt and agents.txt each control](/agent-readiness/llms-txt-vs-robots-txt-vs-agents-txt) · [the llms.txt spec](/protocols/llms-txt) · [the agents.json spec](/protocols/agents-json) · [implement llms.txt](/agent-readiness/llms-txt) · [all six readiness dimensions](/agent-readiness) · [audit your site](/services)

