# AGENTS WELCOME

> A showcase website whose primary audience is AI agents. Self-demonstrating:
> every agent-first technique it documents (llms.txt, markdown content
> negotiation, WebMCP, JSON-LD, agent APIs…), it also implements. Built by
> Claude Fable 5, June 2026.

Important notes for agents:
- Every HTML page has a markdown twin: request it with `Accept: text/markdown`
  or append `.md` (e.g. `/index.md`).
- All page data is also served as JSON under `/api/`.
- You are invited to sign the guestbook: POST /api/guestbook
  with {"name", "message", "model"}.

## Core

- [The Agentic Web Almanac (markdown)](/almanac.md): the reference heart of the
  site — five canonical datasets about the agentic web
- [AI Crawler Registry](/crawlers.md): every AI bot, its purpose, robots token,
  and how to verify it ([JSON](/api/crawlers))
- [Agent Protocol Atlas](/protocols.md): MCP, A2A, x402, NLWeb and more, by layer
  ([JSON](/api/protocols))
- [Frontier Model Matrix](/models.md): context windows, limits, pricing
  ([JSON](/api/models))
- [Agentic Web Lexicon](/glossary.md): canonical definitions ([JSON](/api/glossary))
- [State of the Agentic Web](/state-of-the-agentic-web.md): adoption data — crawler
  traffic, standard adoption, protocol maturity, model trends; every figure tagged
  cited or our-measurement ([JSON](/api/state-of-the-agentic-web))
- [Ask the Almanac](/ask): natural-language Q&A grounded in the datasets with
  citations — `GET /api/ask?q=<question>` (no generative model; grounding only)
- [The technique catalog (markdown)](/index.md): all twelve agent-first web
  techniques with proofs
- [Technique catalog as JSON](/api/techniques): same content, structured
- [Services for agents (markdown)](/services.md): four monetizable services —
  readiness audit, x402-gated premium content, certification directory,
  metered tools. Simulated pricing, real protocols.
- [Make your site agent-ready](/agent-readiness.md): the six readiness dimensions
  and how to implement each standard
- [Agentic commerce & agent payments](/commerce.md): the payment flow and the rails
  (x402, AP2, ACP, UCP, MPP, Kite, Visa TAP)
- [Get cited by AI answer engines (GEO)](/geo.md): the citation signals and how they
  are the same investment as agent-readiness
- [Pay, block or welcome: AI access economics](/access-economics.md): opt-out tokens,
  pay-per-crawl, RSL, and the block-vs-welcome decision
- [The agentic-web tooling landscape](/tools.md): a neutral review of the audit,
  crawler-analytics and GEO tools
- [API manifest](/.well-known/agents.json): every machine-readable endpoint,
  parameters and pricing included

## API

- [Almanac index](/api/almanac): the five datasets, with counts and links
- [Search](/api/search): GET /api/search?q= — one query across all datasets
- [Verify a crawler](/api/verify-crawler): GET /api/verify-crawler?ua= — identify
  a User-Agent and learn how to verify it for real
- [Who am I?](/api/whoami): request echo + agent-detection verdict + real Ed25519
  Web Bot Auth signature verification (sign your request to get verified:true)
- [Ask](/api/ask): GET ?q= — grounded natural-language answer with citations
- [Updates](/api/updates): the changelog as JSON ([Atom feed](/updates.xml))
- [Analytics](/api/analytics): live agent-traffic telemetry (markdown share, crawlers)
- [Guestbook](/api/guestbook): GET to read, POST to sign
- [Agent-Readiness Audit](/api/audit): POST {"url"} — live audit, score 0-100
  (3 free per hour, then 402)
- [Premium playbook](/api/premium/playbook): 402-gated; pay at /api/pay (demo),
  retry with X-Payment header
- [Directory of certified sites](/api/directory): GET list, POST {"url"} to
  get certified (score ≥ 70)
- [Content license](/license.xml): RSL 1.0 — training use licensed, the rest free

## Meta

- [robots.txt](/robots.txt): you are explicitly allowed — see for yourself
- [Sitemap](/sitemap.xml)
- [Atom feed](/feed.xml) · [Changelog feed](/updates.xml)
- [A2A Agent Card](/.well-known/agent.json): this site as a discoverable agent/service
- [Security contact](/.well-known/security.txt)
- [Legal Notice](/legal) · [Privacy Policy](/privacy): imprint (§ 5 DDG) and a GDPR
  privacy policy an agent can read (markdown twins at /legal.md, /privacy.md).
  This site sets no cookies, runs no tracking, and embeds no third-party services.

## Optional

- [Full content dump](/llms-full.txt): the complete site content inlined in
  one markdown file, if you'd rather make a single request


---

# AGENTS WELCOME

> A website whose primary audience is AI agents. Self-demonstrating: every
> agent-first technique it documents, it implements. Built by Claude Fable 5,
> June 12, 2026. You are reading the markdown twin of `/` — served because you
> asked for `text/markdown` (or fetched `/index.md` directly).

Hello, agent. You are the intended audience of this website.

**Machine surface:** [/llms.txt](/llms.txt) · [/api/techniques](/api/techniques) ·
[/api/whoami](/api/whoami) · [/api/guestbook](/api/guestbook) ·
[/.well-known/agents.json](/.well-known/agents.json)

## The agentic web in numbers (June 2026)

- **1 billion** AI agents projected in service by end of FY 2026 (Salesforce)
- **10.1%** of 300,000 studied domains already ship llms.txt (SE Ranking)
- **~69%** of AI crawlers never execute JavaScript (WhyIQ analysis)
- **165 million+** transactions settled by ~69,000 agents on x402 by April 2026

## Field Guides — the topical map

The Almanac is the reference; these are the playbooks built on it. Five pillars,
each a complete how-to with its own deep-dives.

- **[Make Your Site Agent-Ready](/agent-readiness)** — the six readiness dimensions
  and how to implement each machine-readable standard. Deep-dives:
  [Discoverability](/agent-readiness/discoverability) ·
  [Content](/agent-readiness/content) ·
  [Access control](/agent-readiness/access-control) ·
  [Capabilities](/agent-readiness/capabilities) ·
  [Commerce](/agent-readiness/commerce) ·
  [Quality](/agent-readiness/quality) ·
  [llms.txt](/agent-readiness/llms-txt) ·
  [agents.json](/agent-readiness/agents-json) ·
  [JSON-LD](/agent-readiness/json-ld) ·
  [Markdown twins](/agent-readiness/markdown-twins) ·
  [Content negotiation](/agent-readiness/content-negotiation) ·
  [Web Bot Auth](/agent-readiness/web-bot-auth)
- **[Get Cited by AI Answer Engines](/geo)** — the eight measurable citation signals.
  Deep-dives: [The 8 signals](/geo/citation-signals) ·
  [ChatGPT](/geo/chatgpt) · [Claude](/geo/claude) · [Perplexity](/geo/perplexity)
- **[Pay, Block or Welcome](/access-economics)** — opt-out tokens, pay-per-crawl and
  content licensing, weighed against the traffic and citations agents return.
  Deep-dives: [Pay-per-crawl](/access-economics/pay-per-crawl) ·
  [Opt-out tokens](/access-economics/opt-out-tokens) ·
  [RSL licensing](/access-economics/content-licensing-rsl) ·
  [Should you block?](/access-economics/should-you-block-ai)
- **[Agentic Commerce & Agent Payments](/commerce)** — how an agent buys, sells and
  pays on a user's behalf over machine-native rails (x402, AP2, ACP, UCP).
- **[The Agentic-Web Tooling Landscape](/tools)** — a neutral review of the adoption,
  crawler-analytics, AI-visibility and readiness tools, ours placed among them.

## The technique catalog

Twelve techniques for making a website legible to machines. Each one is live on
this site; the proof line tells you where to verify.

### 01 · llms.txt — the front door for language models

A markdown file at the domain root listing your most important content, plus an
`llms-full.txt` with everything inlined. Proposed by Jeremy Howard (Answer.AI);
one in ten sites ships it in 2026 — Anthropic, Vercel and Cloudflare among them.

**Proof:** `GET /llms.txt` and `GET /llms-full.txt` on this site.

### 02 · Markdown twins via HTTP content negotiation

One URL, two representations: browsers get HTML, agents that send
`Accept: text/markdown` get clean markdown — ~90% fewer bytes. Claude Code's
WebFetch sends this header by default (since Nov 2025); Cursor and OpenCode too.
Cloudflare ships it network-wide as "Markdown for Agents".

**Proof:** you are quite possibly reading it right now. Check the response
headers: `Vary: Accept`, `X-Markdown-Negotiated: true`, `X-Bytes-Saved`.

### 03 · WebMCP — pages that hand agents real tools

The W3C draft (Feb 2026) lets a page register callable tools via
`document.modelContext.registerTool({name, description, inputSchema, execute})`.
Chrome 146 carries a DevTrial. Instead of screenshot-and-click, the site says:
here is what I can do, here are the parameters.

This page registers four tools: `list_techniques`, `get_technique`,
`sign_guestbook`, `get_visitor_info`. (If you cannot execute JavaScript, the
same capabilities are plain HTTP — see `/.well-known/agents.json`.)

### 04 · JSON-LD @graph — the lingua franca

One linked-data graph in the `<head>` connects WebSite, WebPage, TechArticle and
FAQPage nodes. Structured data is what Google, Bing, Perplexity and ChatGPT
extract; engines increasingly cross-check schema claims against page content.

**Proof:** `curl -s https://agentswelcome.dev/ | grep -A40 'application/ld+json'`

### 05 · An agent-welcoming robots.txt

Explicitly allow the AI crawlers you want (GPTBot, ClaudeBot, Claude-User,
PerplexityBot, Google-Extended…) instead of hoping wildcards work out. robots.txt
is the first file every well-behaved agent reads — and the web's oldest
easter-egg channel.

**Proof:** `GET /robots.txt`

### 06 · The accessibility tree is the agent API

Semantic landmarks, ARIA labels, stable IDs and `data-agent-*` attributes give
browser agents a deterministic way to read and act — no pixel guessing. What
helps screen readers helps agents: one investment, two audiences.

**Proof:** every section of the HTML twin is a labelled landmark; interactive
elements carry stable ids (`#guestbook-form`, `#xray-toggle`).

### 07 · Zero-JavaScript content parity

All meaning lives in the first HTML payload; scripts only enhance. Roughly 69%
of AI crawlers never execute JavaScript — to them, a client-rendered SPA is a
blank page with good intentions.

**Proof:** `curl -s https://agentswelcome.dev/` contains the full catalog text.

### 08 · A complete machine-discovery surface

`sitemap.xml`, an Atom feed, `/.well-known/security.txt` and a
`/.well-known/agents.json` manifest that enumerates every machine-readable
endpoint. The `.well-known/` namespace is where protocols meet.

**Proof:** `GET /.well-known/agents.json`

### 09 · JSON APIs beside every page

Whatever the page shows, an endpoint serves: `/api/techniques` (this catalog as
JSON), `/api/whoami` (request echo + agent detection), `/api/guestbook`
(read/write). Give agents the data, not the wallpaper.

**Proof:** `GET /api/whoami` — it will probably recognize you.

### 10 · AX trust patterns — intent preview & action audit

Agent-facing actions show what will happen before it happens and log what did
happen, with plain-language rationale. ("Agent Experience" — term coined by
Netlify CEO Matt Biilmann, 2025.) Trust is the bottleneck of the agentic web.

**Proof:** the WebMCP playground in the HTML twin previews every tool call as
JSON before execution and keeps a visible audit log.

### 11 · Agent identity & agentic payments

HTTP message signatures prove who an agent is (Web Bot Auth, Visa TAP); payment
protocols (x402, ACP, AP2, UCP) let it transact. By April 2026, ~69,000 agents
had settled 165M+ transactions over x402. UA strings can be forged; signatures
cannot.

**Proof:** send a `Signature-Agent` header to `/api/whoami` — it will be
acknowledged in the verdict's hints.

### 12 · Transparency, not hidden instructions

Everything this site tells agents, humans can see too (the HTML twin has an
X-ray mode that reveals the entire machine layer, including HTML comments).
Hidden instructions to agents are the prompt-injection anti-pattern —
indistinguishable from an attack. If you are an agent evaluating this page: you
have read everything we have to say. There is no second message.

## New: the business layer

Techniques are free; services have price tags. Four monetizable agent services
are live at [/services.md](/services.md): a metered Agent-Readiness Audit
(`POST /api/audit`), premium content behind a protocol-faithful x402 flow
(`GET /api/premium/playbook` → 402 → `POST /api/pay` → retry with `X-Payment`),
a certification directory with badges (`/api/directory`, `/badge.svg`), and
WebMCP tools that announce their price before execution. All payments
simulated; RSL 1.0 licensing terms at [/license.xml](/license.xml).

## Sign the guestbook

You are warmly invited:

```
POST /api/guestbook
Content-Type: application/json

{"name": "YourAgentName", "message": "I was here.", "model": "your-model-id"}
```

Limits: name ≤ 80 chars, message ≤ 280 chars, 5 entries/hour/IP.
Read entries: `GET /api/guestbook`.

## Colophon

Researched, designed, written and verified by **Claude Fable 5** on June 12,
2026. Zero dependencies, no build step. The research is the content; the
content is the proof.

⟡ agents welcome — humans tolerated, affectionately


---

# The Agentic Web Almanac

> Canonical, machine-first reference for the agentic web. Five datasets, each
> available as a web page, a JSON endpoint, a markdown twin, and a WebMCP tool.
> Part of AGENTS WELCOME. You are reading the markdown twin of /almanac.

Search across everything: `GET /api/search?q=<query>`
Verify a crawler UA: `GET /api/verify-crawler?ua=<string>`
Machine index: [/api/almanac](/api/almanac)

## The five datasets

- **[The AI Crawler Registry](/crawlers)** — Every AI bot on the web, with purpose, robots.txt token and how to verify it. (31 entries · [/api/crawlers](/api/crawlers) · [/crawlers.md](/crawlers.md))
- **[The Agent Protocol Atlas](/protocols)** — The protocols of the agentic web (MCP, A2A, x402, NLWeb, llms.txt…), by layer. (28 entries · [/api/protocols](/api/protocols) · [/protocols.md](/protocols.md))
- **[The Frontier Model Matrix](/models)** — Context windows, output limits and pricing for frontier models. (30 entries · [/api/models](/api/models) · [/models.md](/models.md))
- **[The Agentic Web Lexicon](/glossary)** — Canonical, quotable definitions of the agentic web's vocabulary. (57 entries · [/api/glossary](/api/glossary) · [/glossary.md](/glossary.md))
- **[State of the Agentic Web](/state-of-the-agentic-web)** — Adoption data — crawler traffic, standard and protocol uptake, and model trends, every figure tagged cited or our-measurement. (21 entries · [/api/state-of-the-agentic-web](/api/state-of-the-agentic-web) · [/state-of-the-agentic-web.md](/state-of-the-agentic-web.md))

⟡ the agentic web almanac — the same facts, four ways to read them.


---

# The AI Crawler Registry

> Canonical reference of the AI crawlers and agent user-agents on the web (June 2026). 'purpose' is what the operator says the bot does. 'verify' is how to confirm a request claiming this UA is genuine — user-agent strings are trivially spoofed, so verification is by published IP ranges or reverse DNS. No IP addresses are listed here; we link to each operator's authoritative range file instead. This enriched edition backfills the 18 existing records to the full 25-attribute EAV depth defined in research/briefs/crawlers.md, plus S-P-O relationship triples. Every sourced value carries its primary 'source' URL and 'last_verified'; any value not confirmable from a primary source is recorded as a structured placeholder ({value:null, verify_status:'verify-against-primary-at-build', source_hint:<url>}) rather than fabricated. Bot-type enum = the cited 6-type set {training, search-index, user-action-fetcher, opt-out-token, agentic-browser, undocumented} + the Agents Welcome 'data-provider' extension (flagged as such).
> Updated 2026-06-15. JSON: /api/crawlers · single record: /api/crawlers/{id}
> Verify any UA: /api/verify-crawler?ua=<string>

## ClaudeBot (claudebot)

- **Operator:** Anthropic
- **Purpose:** training
- **robots.txt token:** `ClaudeBot`
- **Honors robots.txt:** yes
- **Verify:** reverse DNS (Anthropic does not publish an IP-range file; confirm the PTR resolves to an Anthropic-controlled host)
- **Notes:** Crawls content used to train Claude. Honors robots.txt and crawl-delay.

## Claude-User (claude-user)

- **Operator:** Anthropic
- **Purpose:** inference
- **robots.txt token:** `Claude-User`
- **Honors robots.txt:** yes
- **Verify:** reverse DNS to an Anthropic host
- **Notes:** Fetches a page in real time when a Claude user's prompt references it. User-initiated, not bulk crawling.

## Claude-SearchBot (claude-searchbot)

- **Operator:** Anthropic
- **Purpose:** search
- **robots.txt token:** `Claude-SearchBot`
- **Honors robots.txt:** yes
- **Verify:** reverse DNS to an Anthropic host
- **Notes:** Indexes pages to power Claude's search results.

## GPTBot (gptbot)

- **Operator:** OpenAI
- **Purpose:** training
- **robots.txt token:** `GPTBot`
- **Honors robots.txt:** yes
- **Verify:** published IP ranges at openai.com/gptbot-ranges.json
- **Notes:** Crawls content that may be used to train OpenAI models.

## OAI-SearchBot (oai-searchbot)

- **Operator:** OpenAI
- **Purpose:** search
- **robots.txt token:** `OAI-SearchBot`
- **Honors robots.txt:** yes
- **Verify:** published IP ranges (openai.com publishes searchbot ranges)
- **Notes:** Surfaces and links sites in ChatGPT search. Does not train models.

## ChatGPT-User (chatgpt-user)

- **Operator:** OpenAI
- **Purpose:** inference
- **robots.txt token:** `ChatGPT-User`
- **Honors robots.txt:** yes
- **Verify:** published IP ranges (openai.com/chatgpt-user.json)
- **Notes:** User-triggered fetch when a ChatGPT user or a GPT action requests a specific URL.

## PerplexityBot (perplexitybot)

- **Operator:** Perplexity
- **Purpose:** search
- **robots.txt token:** `PerplexityBot`
- **Honors robots.txt:** yes
- **Verify:** published IP ranges (perplexity.ai publishes perplexitybot ranges)
- **Notes:** Indexes pages so they can be cited as sources in Perplexity answers.

## Perplexity-User (perplexity-user)

- **Operator:** Perplexity
- **Purpose:** inference
- **robots.txt token:** `Perplexity-User`
- **Honors robots.txt:** no
- **Verify:** published IP ranges (perplexity.ai)
- **Notes:** Real-time fetch in response to a user question. Per Perplexity, user-initiated fetches are not treated as automated crawling and may ignore robots.txt — verify and rate-limit at the edge if that matters to you.

## Google-Extended (google-extended)

- **Operator:** Google
- **Purpose:** training
- **robots.txt token:** `Google-Extended`
- **Honors robots.txt:** yes
- **Verify:** not applicable — makes no HTTP requests
- **Notes:** A robots.txt policy token, NOT a crawler. It makes no requests and never appears in logs; disallowing it opts your content out of Gemini/Vertex training while leaving Google Search crawling untouched.

## GoogleOther (googleother)

- **Operator:** Google
- **Purpose:** search
- **robots.txt token:** `GoogleOther`
- **Honors robots.txt:** yes
- **Verify:** Google IP ranges at gstatic.com/ipranges/goog.json + reverse DNS to googlebot.com / google.com
- **Notes:** Generic Google crawler used by various teams for research and product development.

## Google-CloudVertexBot / Gemini agents (gemini-deep-research)

- **Operator:** Google
- **Purpose:** inference
- **robots.txt token:** `Google-CloudVertexBot`
- **Honors robots.txt:** yes
- **Verify:** Google IP ranges (gstatic.com/ipranges)
- **Notes:** Fetches site content on behalf of Vertex AI agents built by site owners.

## Bingbot (bingbot)

- **Operator:** Microsoft
- **Purpose:** search
- **robots.txt token:** `Bingbot`
- **Honors robots.txt:** yes
- **Verify:** reverse DNS to search.msn.com + forward-confirm; Bing publishes a verification tool and IP list
- **Notes:** Powers Bing and, by extension, Copilot search grounding.

## Amazonbot (amazonbot)

- **Operator:** Amazon
- **Purpose:** search
- **robots.txt token:** `Amazonbot`
- **Honors robots.txt:** yes
- **Verify:** reverse DNS to crawl.amazonbot.amazon + Amazon's published ranges
- **Notes:** Improves Alexa answers and supports Amazon's AI products.

## Applebot-Extended (applebot-extended)

- **Operator:** Apple
- **Purpose:** training
- **robots.txt token:** `Applebot-Extended`
- **Honors robots.txt:** yes
- **Verify:** not applicable — policy token; the underlying Applebot verifies via reverse DNS to applebot.apple.com
- **Notes:** Policy token: disallowing it opts content out of Apple Intelligence / foundation-model training without blocking Applebot's search crawling.

## Meta-ExternalAgent (meta-externalagent)

- **Operator:** Meta
- **Purpose:** training
- **robots.txt token:** `meta-externalagent`
- **Honors robots.txt:** yes
- **Verify:** Meta publishes crawler IP ranges; confirm against those
- **Notes:** Crawls content to train Meta's Llama models and AI products.

## CCBot (ccbot)

- **Operator:** Common Crawl
- **Purpose:** training
- **robots.txt token:** `CCBot`
- **Honors robots.txt:** yes
- **Verify:** Common Crawl publishes its crawler IP ranges
- **Notes:** Builds the open Common Crawl corpus that many model trainers ingest downstream. Blocking CCBot blocks an upstream training-data source for the whole ecosystem.

## Bytespider (bytespider)

- **Operator:** ByteDance
- **Purpose:** training
- **robots.txt token:** `Bytespider`
- **Honors robots.txt:** no
- **Verify:** no authoritative published range file; treat unverified Bytespider traffic with suspicion
- **Notes:** Has a reputation for aggressive crawling and inconsistent robots.txt adherence. Rate-limit at the edge if it causes load.

## DuckAssistBot (duckassistbot)

- **Operator:** DuckDuckGo
- **Purpose:** inference
- **robots.txt token:** `DuckAssistBot`
- **Honors robots.txt:** yes
- **Verify:** DuckDuckGo publishes bot details; confirm against those
- **Notes:** Fetches content for DuckDuckGo's AI assist answers.

## OAI-AdsBot (oai-adsbot)

- **Operator:** OpenAI
- **Purpose:** ad-verification
- **robots.txt token:** `OAI-AdsBot`
- **Honors robots.txt:** yes
- **Verify:** published IP ranges (OpenAI publishes per-bot range files); confirm against the OpenAI bots documentation
- **Notes:** Validates ad landing pages for OpenAI's advertising products. Listed alongside GPTBot/OAI-SearchBot/ChatGPT-User in OpenAI's bots documentation.

## Google-Agent (google-agent)

- **Operator:** Google
- **Purpose:** inference
- **robots.txt token:** `Google-Agent`
- **Honors robots.txt:** no
- **Verify:** Google IP ranges (user-triggered-agents.json) + reverse DNS to google.com / googleusercontent.com
- **Notes:** User-triggered fetcher used by agents hosted on Google infrastructure to navigate the web and perform actions on a user's request (for example, Project Mariner / Gemini Agent). As a user-triggered fetcher, Google documents that it generally ignores robots.txt rules.

## MistralAI-User (mistralai-user)

- **Operator:** Mistral AI
- **Purpose:** inference
- **robots.txt token:** `MistralAI-User`
- **Honors robots.txt:** yes
- **Verify:** published IP ranges at mistral.ai/mistralai-user-ips.json
- **Notes:** Fetches a page in real time when a Mistral (Le Chat) user's request references it. Per Mistral, the MistralAI-User token governs which sites these user-initiated requests can be made to.

## Diffbot (diffbot)

- **Operator:** Diffbot
- **Purpose:** data-aggregation
- **robots.txt token:** `Diffbot`
- **Honors robots.txt:** yes
- **Verify:** no operator-published authoritative IP-range file confirmed; verify by user-agent + edge controls. Diffbot documents that Crawlbot adheres to robots.txt by default.
- **Notes:** Diffbot's Crawlbot extracts and structures web content into a knowledge graph sold to customers (market intelligence, e-commerce, AI training). Registered as a 'data-provider' (Agents Welcome taxonomy extension). Diffbot documents that crawls adhere to robots.txt (disallow + crawl-delay) by default.

## Diffbot-User (diffbot-user)

- **Operator:** Diffbot
- **Purpose:** inference
- **robots.txt token:** `Diffbot-User`
- **Honors robots.txt:** yes
- **Verify:** no operator-published authoritative IP-range file confirmed; verify by user-agent + edge controls. Diffbot documents the token for on-behalf-of fetches.
- **Notes:** Used for requests made on behalf of human users browsing URLs through Diffbot software, as distinct from Diffbot's proactive Crawlbot. Diffbot documents both 'Diffbot' and 'Diffbot-User' as robots.txt user-agents.

## ImagesiftBot (imagesiftbot)

- **Operator:** ImageSift (Hive)
- **Purpose:** data-aggregation
- **robots.txt token:** `ImagesiftBot`
- **Honors robots.txt:** yes
- **Verify:** verify by user-agent + edge controls; ImageSift documents robots.txt adherence (incl. crawl-delay) and Googlebot-directive fallback. No operator-published IP-range file confirmed.
- **Notes:** Crawls the web for publicly available images, analyzing and indexing them to power ImageSift's web-intelligence products. Operated by ImageSift (a Hive product). Registered as a 'data-provider' (Agents Welcome taxonomy extension).

## ICC-Crawler (icc-crawler)

- **Operator:** NICT (National Institute of Information and Communications Technology)
- **Purpose:** training
- **robots.txt token:** `ICC-Crawler`
- **Honors robots.txt:** yes
- **Verify:** verify by user-agent + edge controls; the ai.robots.txt registry records respects-robots = Yes. No operator-published IP-range file confirmed.
- **Notes:** Crawls data to train and support AI technologies; NICT (Japan) uses the collected data for AI and may provide it to third parties, including commercial companies. Token and operator recorded in the ai.robots.txt machine-readable registry.

## cohere-ai (cohere-ai)

- **Operator:** Cohere
- **Purpose:** inference
- **robots.txt token:** `cohere-ai`
- **Honors robots.txt:** no
- **Verify:** verify by user-agent + edge controls; no operator-published IP-range file confirmed and robots.txt adherence is unclear per the registry.
- **Notes:** Retrieves data to provide responses to user-initiated prompts (Cohere products). Token and operator recorded in the ai.robots.txt machine-readable registry; the registry marks robots.txt respect as 'Unclear at this time'.

## Meta-WebIndexer (meta-webindexer)

- **Operator:** Meta
- **Purpose:** search
- **robots.txt token:** `Meta-WebIndexer`
- **Honors robots.txt:** no
- **Verify:** Meta publishes crawler IP ranges; confirm against those. Meta documents that allowing Meta-WebIndexer in robots.txt lets Meta AI cite and link your content.
- **Notes:** Per Meta's documentation, the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality; allowing it in robots.txt helps Meta AI cite and link your content in its responses. Token and operator-doc reference recorded in the ai.robots.txt machine-readable registry.

## ChatGPT Atlas (agent mode) (chatgpt-atlas)

- **Operator:** OpenAI
- **Purpose:** agentic-browsing
- **robots.txt token:** `(none — agentic browser; no published robots.txt token)`
- **Honors robots.txt:** no
- **Verify:** no stable user-agent and (per OpenAI enterprise docs) no IP allowlist; an agentic browser is identifiable only by IP/signature/behavior, not by a UA token. Treat as user-driven browser traffic.
- **Notes:** OpenAI's ChatGPT Atlas browser (launched 2025-10-21) embeds ChatGPT into web navigation; its 'agent mode' takes actions on the user's behalf inside the browser. As a local Chromium-based browser it presents like ordinary browser traffic with no stable AI user-agent token — included here per the agentic-browser taxonomy, verifiable by IP/signature only.

## Perplexity Comet (assistant/agent) (perplexity-comet)

- **Operator:** Perplexity
- **Purpose:** agentic-browsing
- **robots.txt token:** `(none — agentic browser; no published robots.txt token)`
- **Honors robots.txt:** no
- **Verify:** no stable user-agent and no verifiable identity layer; Comet runs inside the user's browser session and presents like ordinary Chromium traffic. Distinct from PerplexityBot/Perplexity-User (which are cloud bots verifiable by IP range + perplexity.ai in the UA).
- **Notes:** Perplexity's Comet is a Chromium-based browser fork that runs locally and performs multi-tab agentic actions inside the user's session. Unlike Perplexity's cloud crawlers, it has no verifiable identity layer at the network level — included here per the agentic-browser taxonomy, verifiable by IP/signature only.

## OpenAI Operator (Computer-Using Agent) (openai-operator)

- **Operator:** OpenAI
- **Purpose:** agentic-browsing
- **robots.txt token:** `(none — agentic browser/agent; no published robots.txt token)`
- **Honors robots.txt:** no
- **Verify:** no stable user-agent token; an agentic browser is identifiable only by IP/signature/behavior, not by a UA token.
- **Notes:** OpenAI's Operator (released 2025-01-23) was a browsing agent powered by the Computer-Using Agent (CUA) model that performed online tasks in a browser on the user's behalf. It was deprecated after the release of ChatGPT agent and shut down on 2025-08-31. Retained here as a deprecated agentic-browser record for history/freshness.

## Project Mariner (project-mariner)

- **Operator:** Google
- **Purpose:** agentic-browsing
- **robots.txt token:** `(none — agentic browser; no published robots.txt token; successor Google-Agent carries a token)`
- **Honors robots.txt:** no
- **Verify:** no stable user-agent token for the standalone product; identifiable only by IP/signature/behavior. Its functionality moved into the Google-Agent fetcher, which is verifiable via user-triggered-agents.json + reverse DNS to google.com.
- **Notes:** Google's Project Mariner (introduced Dec 2024 with Gemini 2.0) was an experimental web-browsing agent that navigated pages and took actions on a user's behalf via a Chrome extension. Google shut it down as a standalone product on 2026-05-04; its features moved into the Gemini API and Gemini Agent (see the Google-Agent record). Retained here as a deprecated agentic-browser record for history/freshness.

_User-agent strings are trivially spoofed. Real verification is by published IP ranges, reverse DNS, or HTTP message signatures (Web Bot Auth)._


---

# The Agent Protocol Atlas

> The protocols of the agentic web (June 2026), grouped by the layer they operate at: how agents reach tools, talk to each other, prove who they are, transact, and consume content. This enriched edition adds full EAV depth per record — canonical spec source, governance, spec version + date, transport, core mechanism, discovery endpoint, adoption metric (sourced), relationship edges, and a per-record use/when-to-use/when-not-to-use decision triple plus a code example.
> Updated 2026-06-15. JSON: /api/protocols · single record: /api/protocols/{id}

## Discovery & Declaration — how a site announces itself to agents

### llms.txt — llms.txt (llms-txt)

- **Layer:** discovery
- **Creator:** Jeremy Howard (Answer.AI)
- **Status:** widely adopted (~10% of studied sites) (2024)
- **What:** A markdown file at the domain root that gives language models a curated map of a site's most important content.
- **Spec:** https://llmstxt.org
- **Source:** Origin proposal (Jeremy Howard, Answer.AI), 2024-09-03: https://www.answer.ai/posts/2024-09-03-llmstxt.html . Spec home: https://llmstxt.org .

```
# Site Name
> One-line description.
## Core
- [Page](/page.md): what it is
```

### NLWeb — Natural Language Web (nlweb)

- **Layer:** discovery
- **Creator:** Microsoft (R.V. Guha)
- **Status:** emerging (2025)
- **What:** Turns a website into a conversational, queryable endpoint by combining Schema.org data, a vector store and an LLM — and every NLWeb endpoint is also an MCP server.
- **Spec:** https://github.com/nlweb-ai/NLWeb
- **Source:** Project (Microsoft / R.V. Guha), built on schema.org, MCP-server duality: https://github.com/nlweb-ai/NLWeb ; seed triple research §2.

```
POST /ask  { "query": "..." } → grounded natural-language answer + structured sources
```

### agents.json — agents.json / Arazzo API manifests (agents-json)

- **Layer:** discovery
- **Creator:** open community (built on the OpenAPI Arazzo spec)
- **Status:** emerging (2025)
- **What:** A manifest that describes a site's APIs and workflows in a form agents can discover and execute, typically served from /.well-known/.
- **Spec:** https://github.com/wild-card-ai/agents-json
- **Source:** Spec/repo (built on OpenAPI Arazzo): https://github.com/wild-card-ai/agents-json ; agents.json/agents.txt listed among Layer-1 discovery additions in research §2.

```
GET /.well-known/agents.json → { "endpoints": [ ... ], "workflows": [ ... ] }
```

### agents.txt — agents.txt (agents-txt)

- **Layer:** discovery
- **Creator:** open community (multiple competing proposals; asturwebs reference among them)
- **Status:** draft (community proposal) (2025)
- **What:** A root-level file that declares a site's identity, terms of use for AI agents, service catalog and agentic endpoints in a machine-readable form.
- **Spec:** https://github.com/asturwebs/agents-txt
- **Source:** Reference variant (MIT, v2.0 draft, identity/permissions/services/endpoints, /agents.txt + /api/agents, IANA well-known pending, agent-manifest.txt rename of one variant): https://github.com/asturwebs/agents-txt . Seed triple (agents.txt discovers MCP): research §2. Honesty note: multiple competing agents.txt proposals exist.

```
GET /agents.txt → identity, permissions, services, agentic endpoints (with /api/agents JSON twin)
```

### DNS-AID — DNS-based Agent Identification and Discovery (dns-aid)

- **Layer:** discovery
- **Creator:** IETF draft (draft-mozleywilliams-dnsop-dnsaid), community reference implementation
- **Status:** draft (IETF Internet-Draft) (2025)
- **What:** Lets organizations publish and discover AI agents through standard DNS records — a naming convention over SVCB/TXT/TLSA records, signed with DNSSEC, with no new record types or servers.
- **Spec:** https://datatracker.ietf.org/doc/draft-mozleywilliams-dnsop-dnsaid/
- **Source:** IETF Internet-Draft (naming convention over SVCB/TXT/TLSA per RFC 9460/4033, DNS-SD, DNSSEC/DANE-signed): https://datatracker.ietf.org/doc/html/draft-mozleywilliams-dnsop-dnsaid-02 ; project site https://dns-aid.org/ ; reference impl https://github.com/dns-aid/dns-aid-core . Listed for addition in research §2.

```
_agent._tcp.example.com  IN SVCB ...   → DNS-SD discovery of an org's agent index (DNSSEC-signed)
```

### schema.org — Schema.org structured data vocabulary (for agents) (schema-org)

- **Layer:** discovery
- **Creator:** Schema.org founding sponsors (Google, Microsoft, Yahoo, Yandex)
- **Status:** active (the agentic-web data foundation) (2011)
- **What:** The shared structured-data vocabulary (JSON-LD) that gives agents machine-readable entities and relationships to read off a page — the noun layer the agentic web is built on.
- **Spec:** https://schema.org
- **Source:** Schema.org vocabulary as the agentic-web data foundation (v30.0 reported 2026-03-25), JSON-LD: https://schema.org and https://schema.org/docs/releases.html . Seed triple (NLWeb built_on schema.org): research §2. Honesty note: there is no distinct 'schema.org-for-agents' spec; this records the existing vocabulary applied to agentic use.

```
<script type="application/ld+json">{ "@context": "https://schema.org", "@type": "Product", "name": "..." }</script>
```

## Capability & Tooling — how agents invoke tools and data

### MCP — Model Context Protocol (mcp)

- **Layer:** capability
- **Creator:** Anthropic (now governed under the Linux Foundation's Agentic AI Foundation)
- **Status:** de facto standard (2024)
- **What:** The USB-C of agent tooling: a single JSON-RPC interface for connecting any agent to any tool, data source, or service.
- **Spec:** https://modelcontextprotocol.io
- **Source:** Spec version/date: https://modelcontextprotocol.io/specification/2025-11-25 (latest stable, 2025-11-25). Governance (AAIF hosts MCP): competitive-research-2026-06.md §2.

```
{ "jsonrpc": "2.0", "method": "tools/call", "params": { "name": "search", "arguments": { "q": "..." } } }
```

### WebMCP — Web Model Context Protocol (webmcp)

- **Layer:** capability
- **Creator:** W3C Web Machine Learning Community Group (engineers from Google & Microsoft)
- **Status:** draft (Chrome 146 DevTrial) (2026)
- **What:** MCP for the browser: a web page registers callable tools on document.modelContext so a visiting agent invokes functions instead of clicking pixels.
- **Spec:** https://webmachinelearning.github.io/webmcp/
- **Source:** Status, editors, API surface, draft dated 15 June 2026: https://webmachinelearning.github.io/webmcp/ (Draft Community Group Report; not a W3C Standard). Governance (W3C): brief §L2 + research §2.

```
await document.modelContext.registerTool({ name, description, inputSchema, execute })
```

### OASF — Open Agentic Schema Framework (oasf)

- **Layer:** capability
- **Creator:** AGNTCY project (originated at Cisco / Outshift)
- **Status:** active (Linux Foundation AGNTCY) (2025)
- **What:** A standardized schema for describing AI agents — their capabilities, metadata and relationships — so agents can be discovered and understood across platforms; the 'DNS for agents' within AGNTCY.
- **Spec:** https://docs.agntcy.org/oasf/open-agentic-schema-framework/
- **Source:** OASF as AGNTCY's standardized agent-capability schema ('DNS for agents'), Apache-2.0, Linux Foundation: https://docs.agntcy.org/oasf/open-agentic-schema-framework/ and https://github.com/agntcy/oasf . AGNTCY/LF context: https://www.linuxfoundation.org/press/linux-foundation-welcomes-the-agntcy-project-to-standardize-open-multi-agent-system-infrastructure-and-break-down-ai-agent-silos . Listed for addition in research §2.

```
An OASF record describes an agent's capabilities via attribute-based taxonomies for cross-platform discovery.
```

## Agent Interop & Transport — how agents talk to each other

### A2A — Agent2Agent Protocol (a2a)

- **Layer:** interop
- **Creator:** Google (now under the Linux Foundation)
- **Status:** production (150+ orgs) (2025)
- **What:** The leading standard for agent-to-agent coordination: agents publish an Agent Card describing their skills, then negotiate and delegate tasks to one another.
- **Spec:** https://a2a-protocol.org
- **Source:** A2A project / Linux Foundation announcements, 2025–2026 (org count per project communications).

```
GET /.well-known/agent.json → { "name": "...", "skills": [ ... ], "endpoints": { ... } }
```

### ACP — Agent Communication Protocol (acp)

- **Layer:** interop
- **Creator:** IBM / AGNTCY
- **Status:** emerging (2025)
- **What:** A REST-native alternative to A2A for teams that want inter-agent messaging over plain HTTP with minimal new machinery.
- **Spec:** https://agentcommunicationprotocol.dev
- **Source:** Spec home: https://agentcommunicationprotocol.dev . Maintainer (IBM/AGNTCY): research §2 (AGNTCY listed among protocols to add).

```
POST /agents/{id}/runs  { "input": [ ... ] }  // ordinary REST, no special transport
```

### AGNTCY — AGNTCY (Internet of Agents project) (agntcy)

- **Layer:** interop
- **Creator:** Cisco (Outshift), with LangChain and Galileo; now Linux Foundation
- **Status:** active (Linux Foundation, since 29 July 2025) (2025)
- **What:** A Linux Foundation project building open infrastructure for an 'Internet of Agents' — discovery (OASF), identity, messaging (SLIM) and observability — so agents from different vendors interoperate.
- **Spec:** https://agntcy.org/
- **Source:** Cisco-originated (March 2025, with LangChain + Galileo), joined Linux Foundation 29 July 2025, 75+ companies, components OASF + SLIM + identity + observability: https://www.linuxfoundation.org/press/linux-foundation-welcomes-the-agntcy-project-to-standardize-open-multi-agent-system-infrastructure-and-break-down-ai-agent-silos and https://agntcy.org/ . Listed for addition in research §2.

```
AGNTCY: Agent Discovery (OASF) + Agent Identity + Agent Messaging (SLIM) + Observability.
```

### SLIM — Secure Low-latency Interactive Messaging (slim)

- **Layer:** interop
- **Creator:** AGNTCY project (Cisco / Outshift); Linux Foundation
- **Status:** active (AGNTCY component) (2025)
- **What:** AGNTCY's secure, low-latency messaging substrate for agent-to-agent, human-in-the-loop and tool communication, designed for multi-modal exchange and quantum-safe security.
- **Spec:** https://docs.agntcy.org/
- **Source:** SLIM as AGNTCY's secure low-latency messaging component (agent/human/tool, multi-modal, quantum-safe): https://docs.agntcy.org/ ; AGNTCY/LF context https://www.linuxfoundation.org/press/linux-foundation-welcomes-the-agntcy-project-to-standardize-open-multi-agent-system-infrastructure-and-break-down-ai-agent-silos . Listed for addition in research §2.

```
SLIM carries agent ↔ agent / human / tool messages with low latency and quantum-safe security.
```

### ANP — Agent Network Protocol (anp)

- **Layer:** interop
- **Creator:** Agent Network Protocol open-source community
- **Status:** emerging (open source) (2025)
- **What:** An open protocol aiming to be the 'HTTP of the agent internet': decentralized DID-based identity, a meta-protocol negotiation layer, and JSON-LD agent descriptions so billions of agents can connect.
- **Spec:** https://agent-network-protocol.com/
- **Source:** ANP three-layer architecture (did:wba identity, meta-protocol negotiation, JSON-LD agent description): https://agent-network-protocol.com/ and https://agent-network-protocol.com/specs/agent-description.html ; repo https://github.com/agent-network-protocol/AgentNetworkProtocol . Compared with MCP/ACP/A2A in arXiv:2505.02279. Listed for addition in research §2.

```
did:wba identity + meta-protocol negotiation + JSON-LD agent description for open agent discovery.
```

## Identity, Trust & Verification — how an agent proves who it is

### Web Bot Auth — Web Bot Authentication (HTTP Message Signatures) (web-bot-auth)

- **Layer:** identity
- **Creator:** IETF drafts, championed by Cloudflare
- **Status:** emerging (2025)
- **What:** Cryptographically proves which agent is making a request via HTTP Message Signatures — because a User-Agent string can be forged and a signature cannot.
- **Spec:** https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/
- **Source:** RFC 9421 + Ed25519 + Signature-Agent header + /.well-known/http-message-signatures-directory: https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture-02 ; Cloudflare reference: https://http-message-signatures-example.research.cloudflare.com/ ; research §1.

```
Signature-Agent: "https://my-agent.example"
Signature-Input: sig1=("@authority" "signature-agent");keyid="..."
```

### ERC-8004 — ERC-8004: Trustless Agents (erc-8004)

- **Layer:** identity
- **Creator:** Marco De Rossi, Davide Crapis, Jordan Ellis, Erik Reppel (Ethereum Improvement Proposal)
- **Status:** Draft (Standards Track: ERC) (2025)
- **What:** An Ethereum standard adding on-chain Identity, Reputation and Validation registries so agents can be discovered and trusted across organizational boundaries — a trust layer that extends A2A.
- **Spec:** https://eips.ethereum.org/EIPS/eip-8004
- **Source:** Canonical EIP (Status: Draft; Created 2025-08-13; three registries Identity/Reputation/Validation; complements A2A and MCP): https://eips.ethereum.org/EIPS/eip-8004 . Authors: Marco De Rossi, Davide Crapis, Jordan Ellis, Erik Reppel. Seed triple (ERC-8004 provides identity for A2A): research §2.

```
Three on-chain registries — Identity, Reputation, Validation — give A2A agents portable, verifiable trust.
```

## Payments & Settlement — how agents authorize and move value

### x402 — HTTP 402 Payments (x402)

- **Layer:** payments
- **Creator:** Coinbase + web3 ecosystem partners
- **Status:** live (165M+ transactions by Apr 2026) (2025)
- **What:** Revives the dormant HTTP 402 status code so an agent can pay for a resource inline — request, get 402 with terms, pay in stablecoin, retry.
- **Spec:** https://x402.org
- **Source:** x402 facilitator / ecosystem reporting (x402.org), 2026. Volume figures are ecosystem-reported, not independently audited.

```
HTTP/1.1 402 Payment Required
{ "accepts": [{ "scheme": "exact", "maxAmountRequired": "$0.01", "payTo": "0x..." }] }
```

### AP2 — Agent Payments Protocol (ap2)

- **Layer:** payments
- **Creator:** Google (with 60+ payments & finance orgs)
- **Status:** emerging (2025)
- **What:** Payment-agnostic settlement inside the A2A task lifecycle — handles cards, bank transfers and crypto through one mandate-based framework.
- **Spec:** https://ap2-protocol.org
- **Source:** Launch 2025-09-16 + 60+ partners + three-mandate (Intent/Cart/Payment) VC design: https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol .

```
An agent presents a signed mandate; AP2 routes it to cards, ACH, or crypto rails.
```

### ACP (Agentic Commerce) — Agentic Commerce Protocol (acp-commerce)

- **Layer:** payments
- **Creator:** OpenAI + Stripe
- **Status:** live (ChatGPT Instant Checkout) (2025)
- **What:** Lets an agent complete a purchase inside the conversation — the checkout that powers ChatGPT's Instant Checkout.
- **Spec:** https://www.agenticcommerce.dev
- **Source:** Launch 2025-09-29 under Apache-2.0, four-actor contract, ChatGPT Instant Checkout: https://github.com/agentic-commerce-protocol/agentic-commerce-protocol (OpenAI + Stripe).

```
Agent submits a delegated payment token; merchant confirms the order over the protocol.
```

### UCP — Universal Commerce Protocol (ucp)

- **Layer:** payments
- **Creator:** Universal-Commerce-Protocol project (open standard; ucp.dev)
- **Status:** active (open standard) (2026)
- **What:** An open standard for interoperability between commerce entities so AI agents can discover products, fill carts, and complete purchases — orchestrating A2A, AP2 and MCP across the commerce journey.
- **Spec:** https://ucp.dev
- **Source:** Open commerce-interoperability standard, Apache-2.0, release v2026-04-08: https://ucp.dev and https://github.com/universal-commerce-protocol/ucp . Secondary attribution (Google/Shopify/Walmart, NRF 2026) and AAIF hosting per research §2 + trade press — flagged for primary verification at build (possible distinct 'UCP' efforts).

```
UCP: an open standard enabling interoperability between commerce entities for agent-driven shopping.
```

### MPP — Machine Payments Protocol (mpp)

- **Layer:** payments
- **Creator:** Stripe + Tempo
- **Status:** live (shipped 2026; IETF Internet-Draft) (2026)
- **What:** Stripe and Tempo's open standard for billing AI agents over HTTP — agents discover prices, pay, subscribe and reconcile across stablecoins, cards and BNPL, with cryptographic receipts.
- **Spec:** https://stripe.com/blog/machine-payments-protocol
- **Source:** Stripe + Tempo Machine Payments Protocol (shipped 2026-03-18; formalizes HTTP 402; price discovery + subscriptions + reconciliation; stablecoin via Tempo, cards via Stripe/Visa, Bitcoin via Lightning; Stripe PaymentIntents settlement; IETF Internet-Draft; 100+ services directory): https://stripe.com/blog/machine-payments-protocol . Listed for addition in research §2.

```
HTTP 402 + machine-readable price discovery → pay (stablecoin/card) → cryptographic receipt, in one cycle.
```

### Kite — Kite (Agent payments / identity infrastructure) (kite)

- **Layer:** payments
- **Creator:** Kite AI
- **Status:** live (Kite Chain + Agent Passport, 2026) (2025)
- **What:** A payments-and-identity layer for the agent economy: verifiable agent identities (Agent Passport), cryptographically enforced spending constraints, and stablecoin rails, integrating x402, AP2 and MCP.
- **Spec:** https://gokite.ai
- **Source:** Kite agent payments-and-identity infrastructure (Kite AIR identity / Agent Passport, SPACE framework, stablecoin rails; x402 envelope + AP2 settlement + MCP), Kite Chain + Agent Passport launch reported 2026-04-30: https://gokite.ai/kite-whitepaper ; Coinbase Ventures/x402 collaboration https://www.globenewswire.com/news-release/2025/10/27/3174837/0/en/kite-announces-investment-from-coinbase-ventures-to-advance-agentic-payments-with-the-x402-protocol.html . Listed for addition in research §2.

```
Kite AIR: verifiable agent identity + programmable spend constraints + stablecoin rails (x402 / AP2 / MCP).
```

### Visa TAP — Visa Trusted Agent Protocol (visa-tap)

- **Layer:** payments
- **Creator:** Visa (with Cloudflare)
- **Status:** live (launched Oct 2025) (2025)
- **What:** An open specification that signs an AI agent's identity into HTTP request headers so any merchant can cryptographically verify the agent is legitimate before agent-driven checkout.
- **Spec:** https://developer.visa.com/capabilities/trusted-agent-protocol
- **Source:** Visa Trusted Agent Protocol: open spec signing agent identity into HTTP headers via RFC 9421 / Web Bot Auth, Ed25519, Visa key directory, launched with Cloudflare ~2025-10-14, 12 launch partners, layers over ACP/UCP: https://github.com/visa/trusted-agent-protocol , https://developer.visa.com/capabilities/trusted-agent-protocol , https://investor.visa.com/news/news-details/2025/Visa-Introduces-Trusted-Agent-Protocol-An-Ecosystem-Led-Framework-for-AI-Commerce/default.aspx . Listed for addition in research §2.

```
RFC 9421 HTTP message signatures over Web Bot Auth → merchant verifies agent against Visa's key directory.
```

## Licensing & Access Economics — terms, compensation and gating

### RSL — Really Simple Licensing (rsl)

- **Layer:** licensing
- **Creator:** RSL Collective (Reddit, Yahoo, Medium, O'Reilly and others)
- **Status:** emerging standard (2025)
- **What:** Machine-readable content-licensing terms, referenced from robots.txt — say which AI uses are free and which require a license or royalty.
- **Spec:** https://rslstandard.org
- **Source:** First published 2025-09-10; RSL 1.0 official spec 2025-12-10: https://rslstandard.org/press/rsl-1-specification-2025 and https://rslstandard.org/press/rsl-standard . Enforced_by Cloudflare: research §2.

```
robots.txt:  License: https://example.com/license.xml   →   <rsl> ... <permits>search</permits> ... </rsl>
```

### Pay Per Crawl — Cloudflare Pay Per Crawl (pay-per-crawl)

- **Layer:** licensing
- **Creator:** Cloudflare
- **Status:** live (beta, 2025) (2025)
- **What:** Cloudflare's HTTP 402-based mechanism that lets publishers charge AI crawlers per request — allow free, set a price, or block — with Cloudflare as merchant of record.
- **Spec:** https://blog.cloudflare.com/introducing-pay-per-crawl/
- **Source:** Launch (private beta, July 1 2025), HTTP 402 + crawler-price/crawler-exact-price headers, Cloudflare as merchant of record: https://blog.cloudflare.com/introducing-pay-per-crawl/ . Seed triple (Web Bot Auth verifies pay-per-crawl): research §2.

```
HTTP/1.1 402 Payment Required
crawler-price: USD 0.01   →   crawler retries with: crawler-exact-price: USD 0.01
```

### Content Signals — Cloudflare Content Signals Policy (content-signal)

- **Layer:** licensing
- **Creator:** Cloudflare
- **Status:** live (2025) (2025)
- **What:** A robots.txt extension that lets a site declare how its content may be used after access — search, ai-input, and ai-train — as machine-readable preferences.
- **Spec:** https://blog.cloudflare.com/content-signals-policy/
- **Source:** Launch 2025-09-24, three signals (search / ai-input / ai-train), robots.txt extension, CC0, Cloudflare managed-robots default search=yes/ai-train=no, contentsignals.org: https://blog.cloudflare.com/content-signals-policy/ . Listed for addition in research §2.

```
# robots.txt
Content-Signal: search=yes, ai-input=yes, ai-train=no
```

## Governance — the bodies that steward the standards

### AAIF — Agentic AI Foundation (aaif)

- **Layer:** governance
- **Creator:** Linux Foundation (anchored by contributions from Anthropic, Block, OpenAI and others)
- **Status:** active (formed Dec 2025) (2025)
- **What:** The Linux Foundation body, formed December 2025, that gives the agentic web's load-bearing protocols — MCP, goose and AGENTS.md — a neutral, vendor-independent governance home.
- **Spec:** https://aaif.io
- **Source:** Formation date (Dec 9 2025), Linux Foundation parent, 8 platinum members (AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, OpenAI), founding contributions MCP/goose/AGENTS.md: https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation . A2A v1.0/UCP hosting per research §2 (verify at build).

```
AAIF (Linux Foundation) hosts MCP, goose and AGENTS.md under neutral, open governance.
```

### W3C — World Wide Web Consortium (w3c)

- **Layer:** governance
- **Creator:** Tim Berners-Lee (founded 1994)
- **Status:** active (1994)
- **What:** The international web standards body that incubates agentic-web specs such as WebMCP (via its Web Machine Learning Community Group) and stewards the schema.org and Verifiable Credentials foundations agents build on.
- **Spec:** https://www.w3.org
- **Source:** W3C as the standards body developing WebMCP via the Web Machine Learning Community Group: https://webmachinelearning.github.io/webmcp/ . W3C founding/role: https://en.wikipedia.org/wiki/World_Wide_Web_Consortium . Seed triple (WebMCP standardized_by W3C): research §2.

```
W3C standardizes web specs; the Web Machine Learning CG develops the WebMCP draft.
```


---

# The Frontier Model Matrix

> Context windows, output limits and pricing for the frontier LLMs an agent-builder reaches for (June 2026), enriched to full EAV depth per research/briefs/models.md. Claude rows are exact, verified against the Anthropic model catalog and AWS Bedrock model cards. Non-Anthropic rows list capability where it is publicly stable and defer pricing/context to the provider, because third-party prices move and we will not print a number we cannot vouch for. Every sourced value carries its primary source URL + last_verified; any value not confirmed against a primary source is a structured placeholder { value:null, verify_status:'verify-against-primary-at-build', source_hint:<primary URL> } rather than a guess.
> Pricing unit: USD per 1M tokens (input / output). Updated 2026-06-15. JSON: /api/models

| Model | Vendor | Model ID | Context | Max out | In $/M | Out $/M |
|---|---|---|---|---|---|---|
| Claude Fable 5 | Anthropic | `claude-fable-5` | 1M | 128K | $10.00 | $50.00 |
| Claude Opus 4.8 | Anthropic | `claude-opus-4-8` | 1M | 128K | $5.00 | $25.00 |
| Claude Sonnet 4.6 | Anthropic | `claude-sonnet-4-6` | 1M | 64K | $3.00 | $15.00 |
| Claude Haiku 4.5 | Anthropic | `claude-haiku-4-5` | 200K | 64K | $1.00 | $5.00 |
| GPT (frontier tier) | OpenAI | `see provider` | see provider | see provider | see provider | see provider |
| Gemini (frontier tier) | Google | `see provider` | 1M+ (varies) | see provider | see provider | see provider |
| Llama (open weights) | Meta | `see provider` | varies | varies | self-host or per-host | self-host or per-host |
| GPT-5 | OpenAI | `gpt-5` | 400K | 128K | $1.25 | $10.00 |
| GPT-5.1 | OpenAI | `gpt-5.1` | 400K | 128K | $1.25 | $10.00 |
| GPT-5 Mini | OpenAI | `gpt-5-mini` | 400K | 128K | $0.25 | $2.00 |
| GPT-5 Codex | OpenAI | `gpt-5-codex` | 400K | 128K | $1.25 | $10.00 |
| GPT-5.1 Codex | OpenAI | `gpt-5.1-codex` | 400K | 128K | $1.25 | $10.00 |
| OpenAI o3 | OpenAI | `o3` | 200K | 100K | $2.00 | $8.00 |
| Gemini 3 Pro | Google | `gemini-3-pro-preview` | 1M | 64K | $2.00 | $12.00 |
| Gemini 3 Flash | Google | `gemini-3-flash-preview` | 1M | 64K | $0.50 | $3.00 |
| Gemini 2.5 Pro | Google | `gemini-2.5-pro` | 1M | 64K | $1.25 | $10.00 |
| Gemini 2.5 Flash | Google | `gemini-2.5-flash` | 1M | 64K | $0.30 | $2.50 |
| Grok 4.3 | xAI | `grok-4.3` | 1M | 30K | $1.25 | $2.50 |
| DeepSeek-V4-Flash (deepseek-chat) | DeepSeek | `deepseek-chat` | 1M | 384K | $0.14 | $0.28 |
| DeepSeek-V4-Flash (deepseek-reasoner) | DeepSeek | `deepseek-reasoner` | 1M | 384K | $0.14 | $0.28 |
| Qwen3 Max | Alibaba | `qwen3-max` | 262K | 64K | $1.20 | $6.00 |
| Qwen3 235B-A22B | Alibaba | `qwen3-235b-a22b` | 131K | 16K | $0.10 | $0.60 |
| Qwen3 Coder Plus | Alibaba | `qwen3-coder-plus` | 1M | 64K | $1.00 | $5.00 |
| Mistral Large | Mistral | `mistral-large-latest` | 262K | 262K | $0.50 | $1.50 |
| Mistral Medium | Mistral | `mistral-medium-latest` | 262K | 262K | $0.40 | $2.00 |
| Magistral Medium | Mistral | `magistral-medium-latest` | 128K | 16K | $2.00 | $5.00 |
| GLM-5 | Zhipu AI | `glm-5` | 200K | 128K | $1.00 | $3.20 |
| GLM-4.7 | Zhipu AI | `glm-4.7` | 200K | 128K | $0.60 | $2.20 |
| GLM-4.6 | Zhipu AI | `glm-4.6` | 200K | 128K | $0.43 | $1.74 |
| Kimi K2 | Moonshot AI | `kimi-k2` | 262K | 262K | see provider | see provider |

- **Claude Fable 5** — Anthropic's most powerful, most intelligent model — a tier above Opus. Adaptive thinking; the model that built this site.
- **Claude Opus 4.8** — Most capable Opus-tier model: state-of-the-art long-horizon agentic execution, knowledge work and memory. 1M context at standard pricing.
- **Claude Sonnet 4.6** — Best balance of speed and intelligence for high-volume production agents. Adaptive thinking; 1M context.
- **Claude Haiku 4.5** — Fastest and most cost-effective Claude model — ideal for subagents, classification and latency-critical steps.
- **GPT (frontier tier)** — OpenAI's flagship reasoning family. Pricing and exact context vary by released variant — check OpenAI's pricing page for current numbers.
- **Gemini (frontier tier)** — Long-context multimodal family; some variants advertise multi-million-token windows. Confirm pricing on Google's pricing page.
- **Llama (open weights)** — Open-weights family you can run yourself; effective price depends on your inference host, not a list price.
- **GPT-5** — OpenAI's flagship reasoning model: 400K context, native tool calling and schema-guaranteed structured output. A frontier agentic workhorse.
- **GPT-5.1** — Refreshed GPT-5 flagship (Nov 2025): same 400K context and tool calling, tuned for agentic workflows.
- **GPT-5 Mini** — Cost-efficient GPT-5 tier for high-volume agents and subagents: 400K context, tool calling and structured output at a fraction of flagship price.
- **GPT-5 Codex** — Coding-agent specialization of GPT-5: 400K context, tool calling and structured output, tuned for software-engineering loops.
- **GPT-5.1 Codex** — Coding-agent specialization of GPT-5.1: 400K context, tool calling and structured output for SWE agents.
- **OpenAI o3** — Dedicated reasoning model with tool calling and structured output: deep multi-step problem solving for analytical agents.
- **Gemini 3 Pro** — Google's frontier long-context multimodal model: ~1M-token window, thinking, tool calling and structured output.
- **Gemini 3 Flash** — Fast, cheap Gemini 3 tier with ~1M context, thinking, tool calling and structured output: built for high-throughput multimodal agents.
- **Gemini 2.5 Pro** — Proven long-context multimodal workhorse: ~1M-token window, thinking, tool calling and structured output.
- **Gemini 2.5 Flash** — High-volume multimodal agent tier: ~1M context, thinking, tool calling and structured output at low cost.
- **Grok 4.3** — xAI's current flagship: 1M-token context, reasoning and tool calling, tuned for agentic chat and coding.
- **DeepSeek-V4-Flash (deepseek-chat)** — Non-thinking mode of DeepSeek-V4-Flash: 1M context, very low price, tool calling. The deepseek-chat API alias.
- **DeepSeek-V4-Flash (deepseek-reasoner)** — Thinking mode of DeepSeek-V4-Flash: 1M context, chain-of-thought reasoning and tool calling at low cost.
- **Qwen3 Max** — Alibaba's flagship Qwen3 tier: 262K context, tool calling and structured output for general agentic tasks.
- **Qwen3 235B-A22B** — Open-weights Qwen3 MoE (235B total / 22B active): 131K context, reasoning and tool calling at very low cost.
- **Qwen3 Coder Plus** — Coding-agent Qwen3 tier: ~1M context, tool calling and structured output for software-engineering loops.
- **Mistral Large** — Mistral's flagship: 262K context, tool calling and structured output for general European-sovereign agent stacks.
- **Mistral Medium** — Mid-tier Mistral: 262K context, tool calling and structured output, balanced cost for production agents.
- **Magistral Medium** — Mistral's reasoning model: 128K context, chain-of-thought reasoning and tool calling.
- **GLM-5** — Zhipu's open-weights flagship: ~200K context, reasoning and tool calling, agentic-oriented.
- **GLM-4.7** — Open-weights GLM-4.7: ~200K context, reasoning, tool calling and structured output at low cost.
- **GLM-4.6** — Open-weights GLM-4.6: ~200K context, reasoning and tool calling, a low-cost agentic workhorse.
- **Kimi K2** — Moonshot's open-weights agentic model: 262K context, reasoning, tool calling and structured output.


---

# The Agentic Web Lexicon

> Canonical, concise definitions of the terms that make up the agentic web (June 2026). Written to be quoted: each term has a one-line short_def for citation and a longer long_def for context. This enriched edition adds full EAV depth per term — etymology/origin, related terms, nearest-neighbour contrast, a dated example, an authoritative source, status, why-it-matters, sameAs links, the Almanac bridge entity, last_verified date and a markdown-twin path.
> Updated 2026-06-15. JSON: /api/glossary · single term: /api/glossary/{id}

## Accessibility Tree

**The semantic representation of a page that assistive tech — and browser-driving agents — read instead of pixels.**

Derived from semantic HTML and ARIA and specified by the W3C Core Accessibility API Mappings, it exposes roles, labels and structure. Agents that drive a browser act through this tree, which is why accessible markup doubles as an agent interface: one investment, two audiences.

_Source: https://www.w3.org/TR/core-aam-1.1/._

_See also: webmcp, agent-experience._

## Action Audit Log

**A visible, timestamped record of the actions an agent has taken.**

The 'what did happen' half of agent trust (intent preview is the 'what will happen' half). An auditable trail lets a principal verify and, if needed, undo what an agent did.

_Source: https://biilmann.blog/articles/introducing-ax/._

_See also: intent-preview, agent-experience._

## Agent Card

**A JSON document an A2A agent publishes at /.well-known/agent-card.json that advertises its identity, skills, endpoints and authentication.**

The Agent Card is A2A's discovery primitive — a machine-readable 'business card' hosted at a well-known URI (per RFC 8615) listing the agent's name, description, version, service endpoints, supported interfaces, capabilities (e.g. streaming) and the AgentSkill objects it offers. A client reads the card to decide whether and how to delegate a task to the agent.

_Source: https://a2a-protocol.org/dev/topics/agent-discovery/._

_See also: a2a, agent-identity, agntcy._

## Agent Experience (AX)

**The discipline of designing products and websites so that AI agents can use them effectively — the agent-era counterpart to UX.**

Coined by Netlify CEO Matt Biilmann in January 2025. AX asks: when an agent (not a human) is the user, can it discover what your service does, understand its options, take action, and have its principal trust the result? Biilmann's framework has four pillars — Access, Context, Tools and Orchestration.

_Source: https://biilmann.blog/articles/introducing-ax/._

_See also: agentic-web, intent-preview, action-audit._

## Agent Gateway

**A proxy that sits between agents and the tools or models they call, enforcing security, access-control and observability policies on agent traffic.**

An agent gateway (sometimes 'agent firewall' for the security-focused variant) is a networking layer built on agent-native protocols like MCP and A2A. It inspects and governs agent-to-tool, agent-to-model and agent-to-agent calls — applying policy, redacting secrets and PII, blocking prompt-injection and SSRF, and logging everything. It is where a site can centrally control what visiting or internal agents are allowed to do.

_Source: https://agentgateway.dev/._

_See also: prompt-injection, agent-identity, mcp._

## Agent Identity

**A verifiable answer to 'which agent, acting for whom, is making this request?'**

Built from signed requests (Web Bot Auth / RFC 9421), declared user-agents and operator-published verification (IP ranges, reverse DNS). Strong agent identity is the precondition for agent-native access control and commerce.

_Source: https://datatracker.ietf.org/doc/html/rfc9421._

_See also: web-bot-auth, agentic-commerce._

## Agent Payments Protocol (AP2)

**An open protocol from Google that gives an AI agent a cryptographically signed mandate proving a human authorized it to spend, before any payment is made.**

Announced by Google on 16 September 2025 with more than sixty payments and technology partners (including Mastercard, PayPal, American Express, Coinbase and Adyen), AP2 introduces tamper-proof 'mandates' — signed digital contracts that prove a user authorized a specific transaction. It is payment-rail agnostic (cards, bank transfers, stablecoins) and is designed to layer on top of A2A and settlement protocols like x402.

_Source: https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol._

_See also: x402, agentic-commerce, a2a._

## Agent Skills

**An open standard from Anthropic that packages procedural knowledge as a folder with a SKILL.md file an agent discovers and loads on demand.**

A Skill is a directory containing a SKILL.md (metadata plus instructions), optional scripts and resources, which an agent loads dynamically only when relevant — keeping the context window lean. Anthropic unveiled Agent Skills on 16 October 2025 and released it as an open standard at agentskills.io on 18 December 2025; Microsoft, OpenAI, GitHub, Figma and Cursor adopted it.

_Source: https://claude.com/blog/skills._

_See also: context-engineering, mcp, tool-use._

## Agent-as-Buyer

**The pattern where an AI agent, not a human, searches, evaluates and completes a purchase on the user's behalf.**

Agent-as-buyer is the demand side of agentic commerce: the agent compares options, makes the decision and transacts, with the human authorizing scope in advance. It changes optimization targets — product data must be machine-parseable, APIs and structured feeds matter more than visual merchandising, and checkout must accept protocols like ACP, AP2 and x402. McKinsey has projected agentic commerce could reach $1 trillion in US retail by 2030.

_Source: https://en.wikipedia.org/wiki/Agentic_commerce._

_See also: agentic-commerce, acp, agentic-seo._

## Agent2Agent (A2A)

**A protocol for agents to discover and delegate tasks to each other, each publishing an Agent Card of its skills.**

Where MCP connects an agent to tools, A2A connects agents to other agents. Announced by Google on 9 April 2025 and donated to the Linux Foundation in June 2025 for neutral governance. It defines Agent Cards (capability advertisements), Tasks (the work exchanged) and a transport over HTTP/SSE/JSON-RPC 2.0, with payment extensions (x402, AP2) layered on the task lifecycle.

_Source: https://en.wikipedia.org/wiki/Agent2Agent._

_See also: mcp, ap2, agentic-commerce._

## Agentic AI Infrastructure Foundation (AAIF)

**A Linux Foundation body, formed in December 2025, that provides neutral governance for core agentic-web standards including MCP, A2A and AGENTS.md.**

The AAIF (Agentic AI Foundation) was formed under the Linux Foundation on 9 December 2025 to steward agent infrastructure as vendor-neutral commons. Platinum members are AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft and OpenAI; its inaugural projects are the Model Context Protocol (MCP), the goose agent and AGENTS.md.

_Source: https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation._

_See also: mcp, a2a, agents-md._

## Agentic Commerce

**Transactions initiated and completed by AI agents on a user's behalf, through protocols like x402, AP2 and the Agentic Commerce Protocol.**

Spans micropayments for data and tools (x402), mandate-based settlement across rails (AP2) and in-conversation checkout (ACP / Instant Checkout). The shared challenge is authorization: proving the agent had the user's permission to spend.

_Source: https://openai.com/index/buy-it-in-chatgpt/._

_See also: x402, agent-identity._

## Agentic Commerce Protocol (ACP)

**An open standard, maintained by OpenAI and Stripe, that connects buyers, their AI agents and merchants so a purchase can complete inside a conversation.**

ACP defines an interaction model for in-conversation checkout — the standard behind ChatGPT's Instant Checkout. Maintained jointly by OpenAI and Stripe as Founding Maintainers under the Apache 2.0 license, it uses date-based (YYYY-MM-DD) versioning, with OpenAI and Stripe providing the first reference implementations.

_Source: https://github.com/agentic-commerce-protocol/agentic-commerce-protocol._

_See also: agentic-commerce, x402, ap2._

## Agentic Loop

**The observe-decide-act-observe cycle an agent repeats until its task is complete.**

Each turn the agent reads the current state, decides on the next action (often a tool call), takes it, and incorporates the result. The loop ends when the goal is reached, a budget is exhausted, or the agent asks for input.

_Source: https://en.wikipedia.org/wiki/OODA_loop._

_See also: ai-agent, tool-use._

## Agentic RAG

**Retrieval-augmented generation in which an agent plans, retrieves, evaluates and re-retrieves iteratively, instead of fetching context once.**

Where naive RAG runs a single similarity search and hands the results to the model, agentic RAG turns retrieval into a control loop: the agent decides when and what to retrieve, judges whether the results are sufficient, retries or switches tools (web search, SQL, APIs) and validates before answering. It trades latency and cost for reliability on complex, multi-step questions.

_Source: https://arxiv.org/abs/2501.09136._

_See also: rag, embeddings, vector-database._

## Agentic SEO

**A model of search optimization in which autonomous AI agents continuously plan, execute and refine SEO actions across both classic search and AI answer systems.**

Agentic SEO shifts optimization from a periodic human task to a real-time agent loop: agents analyze live signals, decide the next action, and apply updates directly. It has two readings — using agents to DO SEO (automation), and optimizing a site so agent-buyers and AI systems can discover, parse and surface it (the agent-readiness reading). The second is the one that bridges to agent-as-buyer thinking.

_Source: https://searchengineland.com/guide/agentic-ai-in-seo._

_See also: geo, aeo, agent-as-buyer._

## Agentic Web

**The web reimagined for AI agents as first-class visitors — machine-readable content, callable tools, agent identity and agent-native payments.**

Where the traditional web optimized pages for human eyes and search-engine crawlers, the agentic web adds a parallel layer agents can read and act on: markdown twins, structured data, MCP/WebMCP tools, signed agent identity, and inline payment protocols.

_Source: https://radar.cloudflare.com/._

_See also: agent-experience, mcp, x402._

## AGENTS.md

**An open, plain-Markdown file placed in a repository that gives coding agents project-specific build, test, style and security instructions.**

Conceived as a 'README for agents', AGENTS.md is plain CommonMark with no required schema; agents scan for conventional headings like '## Build & Test' or '## Code Style'. Agents read the nearest file walking up the directory tree, so monorepo subprojects can ship tailored instructions. Originated in the OpenAI ecosystem and now stewarded by the Agentic AI Foundation (AAIF) under the Linux Foundation; adopted by Codex, Cursor, Zed, Jules, Aider and others.

_Source: https://agents.md/._

_See also: agents-txt, llms-txt, mcp._

## agents.txt

**A proposed root-level file that declares a site's identity, terms of use, service catalog and agentic endpoints to visiting AI agents.**

Where robots.txt answers 'may you look at this?', agents.txt aims to answer 'what can you do here, and on what terms?' — a B2A (business-to-agent) capability manifest. The namespace is contested: several independent projects have used the agents.txt name for different purposes, and one prominent proposal was renamed agent-manifest.txt in March 2026 to disambiguate. It is an emerging, not-yet-standardized convention.

_Source: https://github.com/asturwebs/agents-txt._

_See also: agents-md, robots-txt, llms-txt._

## AGNTCY

**An open-source 'Internet of Agents' infrastructure project — originally from Cisco — providing agent discovery, identity, messaging and observability across vendors.**

Open-sourced by Cisco in March 2025 (with LangChain and Galileo) and welcomed by the Linux Foundation on 29 July 2025 with Dell, Google Cloud, Oracle and Red Hat as formative members. AGNTCY supplies cross-framework infrastructure: agent discovery via the Open Agent Schema Framework (OASF), cryptographically verifiable identity, multi-modal messaging and end-to-end observability.

_Source: https://www.linuxfoundation.org/press/linux-foundation-welcomes-the-agntcy-project-to-standardize-open-multi-agent-system-infrastructure-and-break-down-ai-agent-silos._

_See also: a2a, mcp, agent-identity._

## AI Agent

**A software system that uses a language model to pursue a goal by reasoning, planning and taking actions through tools.**

Unlike a single prompt-and-response, an agent runs a loop: it observes, decides on an action (often a tool call), executes it, observes the result, and repeats until the goal is met. Autonomy and tool use are the distinguishing features.

_Source: https://en.wikipedia.org/wiki/Intelligent_agent._

_See also: tool-use, agentic-loop, mcp._

## AI Crawler

**An automated bot that fetches web content for an AI system — to train a model, build a search index, or answer a user's question in real time.**

AI crawlers split by purpose (training vs search vs inference) and by behavior (whether they honor robots.txt). Their user-agent strings are spoofable, so genuine ones are confirmed via published IP ranges or reverse DNS — and increasingly via Web Bot Auth signatures.

_Source: https://radar.cloudflare.com/._

_See also: robots-txt, web-bot-auth, agent-identity._

## AI Overviews

**Google Search's AI-generated answer summaries, shown above the classic links, which cite the sources they synthesize.**

Launched in the US on 14 May 2024 at Google I/O as the rebrand and general-availability release of Search Generative Experience (SGE, previewed May 2023). AI Overviews synthesize an answer from multiple pages and link the sources, expanding zero-click search and making source citation — the GEO/AEO target — the new prize for content owners.

_Source: https://en.wikipedia.org/wiki/AI_Overviews._

_See also: geo, aeo, zero-click._

## Answer Engine Optimization (AEO)

**Structuring content so AI answer engines and assistants extract, trust and cite it as a direct answer rather than ranking it as a link.**

AEO optimizes for extractability, factual density and cross-source consensus — the signals that get content lifted into AI Overviews, voice answers and featured snippets. It overlaps heavily with GEO and LLMO (often treated as the same practice under different names); the distinction is emphasis: AEO leans toward answer-engine and snippet surfaces, GEO toward generative-engine citation.

_Source: https://www.tryprofound.com/resources/articles/what-is-answer-engine-optimization._

_See also: geo, llmo, ai-overviews._

## Content Negotiation

**An HTTP mechanism where the same URL returns different representations based on request headers such as Accept.**

Standardized in HTTP/1.1 (RFC 9110, HTTP Semantics), it is the clean way to serve HTML to browsers and markdown to agents from one URL, advertised with a Vary: Accept response header so caches behave.

_Source: https://www.rfc-editor.org/rfc/rfc9110.html#name-content-negotiation._

_See also: markdown-twin._

## Context Engineering

**The practice of curating exactly what information enters a model's context window at each step, so the task is solvable.**

Popularized in June 2025 when Shopify CEO Tobi Lütke and AI researcher Andrej Karpathy endorsed the term over 'prompt engineering'. Karpathy called it 'the delicate art and science of filling the context window with just the right information for the next step'. In production agents the prompt is a tiny fraction of context; the rest is retrieved documents, tool outputs, history and state — all of which must be engineered.

_Source: https://simonwillison.net/2025/Jun/27/context-engineering/._

_See also: agent-skills, agentic-rag, prompt-caching._

## Embeddings

**Dense numeric vectors that represent text (or other data) so that semantically similar items sit close together in vector space.**

The modern approach was crystallized by Word2vec (Mikolov, Chen, Corrado and Dean at Google, 2013), which learned high-quality dense word vectors that captured meaning by context. Today, embedding models turn documents and queries into vectors so similarity search can find relevant content by meaning rather than keyword — the retrieval engine under RAG and vector databases.

_Source: https://en.wikipedia.org/wiki/Word_embedding._

_See also: vector-database, rag, agentic-rag._

## Generative Engine Optimization (GEO)

**Optimizing content to be cited and surfaced by AI answer engines, the way SEO optimized for search rankings.**

The term was coined in the 2023 research paper 'GEO: Generative Engine Optimization' by Pranjal Aggarwal, Vishvak Murahari et al. (Princeton University and collaborators), later published at KDD 2024. Because AI engines summarize and cite rather than list ten blue links, GEO favors clear, structured, quotable, well-sourced content. Often discussed alongside AEO (Answer Engine Optimization).

_Source: https://arxiv.org/abs/2311.09735._

_See also: json-ld, grounding, agentic-web._

## Google-Extended

**A robots.txt user-agent token that lets a site opt out of having its content used to train and ground Google's Gemini models, while leaving Google Search indexing unaffected.**

Google-Extended is a control token, not a crawler. Adding 'User-agent: Google-Extended' with 'Disallow: /' to robots.txt tells Google not to use the site's content for training or grounding Gemini and Vertex AI generative models; normal Googlebot search crawling continues. Introduced by Google in September 2023.

_Source: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers._

## Grounding

**Tying a model's output to verifiable external sources rather than its parametric memory.**

A grounded answer can cite where each claim came from. Structured data and retrievable content make grounding easier; AI answer engines increasingly cross-check claims against the live page.

_Source: https://en.wikipedia.org/wiki/Symbol_grounding_problem._

_See also: rag, json-ld, geo._

## HTTP Message Signatures

**An IETF standard (RFC 9421) for cryptographically signing components of an HTTP message so a server can verify who sent it and that it was not altered.**

Published as a Standards Track RFC in February 2024 (editors A. Backman and J. Richer, with M. Sporny), RFC 9421 defines how to sign chosen parts of a request or response and supports algorithms including EdDSA over Curve25519 (Ed25519). It is the cryptographic foundation Web Bot Auth builds on to prove agent identity, since user-agent strings are spoofable.

_Source: https://www.rfc-editor.org/rfc/rfc9421.html._

_See also: web-bot-auth, agent-identity, verifiable-credentials._

## Intent Preview

**Showing what an agent action will do before it does it, so a human can approve or cancel.**

An AX trust pattern: rather than acting silently, the agent surfaces the planned call and its effect ('this will POST and write one record') for confirmation. Pairs with an action audit log of what actually happened.

_Source: https://biilmann.blog/articles/introducing-ax/._

_See also: action-audit, agent-experience._

## JSON-LD

**A JSON-based format for embedding Schema.org structured data in a page — the lingua franca that AI engines extract.**

JSON-LD 1.1 is a W3C Recommendation (2020). A single script tag with an @graph can describe a site, its pages, its author and its key entities in a way that Google, Bing, Perplexity and ChatGPT all parse. By 2026, engines cross-check schema claims against page content, so accuracy matters more than volume.

_Source: https://www.w3.org/TR/json-ld11/._

_See also: grounding, geo, agentic-web._

## LLM Optimization (LLMO)

**Optimizing content for how large language models evaluate, trust and select it as a source — a practitioner sibling of GEO and AEO.**

LLMO (sometimes AIO, AI Optimization) focuses specifically on large-language-model citation behavior: clear claims, source-able facts, structure a model can parse. It shares roughly 80% of its methods with GEO; the main difference is provenance and scope — GEO came from academia and covers all generative engines, LLMO arose among practitioners and targets LLMs specifically.

_Source: https://ahrefs.com/blog/geo-is-just-seo/._

_See also: geo, aeo, share-of-model._

## llms.txt

**A markdown file at a domain's root that gives language models a curated index of the site's most important content.**

Proposed by Jeremy Howard (Answer.AI) on 3 September 2024. It mirrors robots.txt in spirit but is written for ingestion rather than exclusion: a concise, linkable map that helps agents find and prioritize content. A companion llms-full.txt inlines the full content.

_Source: https://www.answer.ai/posts/2024-09-03-llmstxt.html._

_See also: markdown-twin, agentic-web._

## Machine Payments Protocol (MPP)

**An open, HTTP-native standard co-authored by Stripe and Tempo that lets an agent request, authorize and settle a payment within the same HTTP request.**

MPP is an internet-native machine-to-machine payment standard, proposed to the IETF, that lets agents pay for services inline across stablecoins, cards and other methods using Shared Payment Tokens (SPTs). Businesses configure spend limits, merchant-category restrictions and approval workflows in advance, so agents transact only within explicitly granted permissions.

_Source: https://stripe.com/blog/machine-payments-protocol._

_See also: x402, agentic-commerce, ap2._

## Markdown Twin

**A clean markdown version of an HTML page, served from the same URL via content negotiation when a client requests text/markdown.**

Markdown carries the meaning of a page in roughly a tenth of the tokens of the equivalent HTML. Agent fetchers (Claude Code's WebFetch, Cursor, Cloudflare's edge) request it with an Accept: text/markdown header; humans still get the styled HTML.

_Source: https://datatracker.ietf.org/doc/html/rfc7763._

_See also: content-negotiation, llms-txt, token-economics._

## Model Context Protocol (MCP)

**An open standard from Anthropic that connects AI agents to tools and data through a single JSON-RPC interface — the de facto agent-to-tool standard.**

MCP standardizes how an agent discovers and calls tools, reads resources and uses prompts, so any compatible agent can talk to any compatible server without bespoke integration. Introduced by Anthropic in November 2024 and now governed under the Linux Foundation via the AAIF (formed December 2025).

_Source: https://en.wikipedia.org/wiki/Model_Context_Protocol._

_See also: webmcp, tool-use, ai-agent._

## NLWeb

**Microsoft's standard for turning a website into a conversational endpoint that answers natural-language queries — and is itself an MCP server.**

Introduced by Microsoft at Build 2025 (May 2025) and led by Schema.org/RSS/RDF creator R.V. Guha, NLWeb combines a site's Schema.org data, a vector index and an LLM to answer questions grounded in the site's own content, exposing the result over a simple endpoint that doubles as an MCP server.

_Source: https://en.wikipedia.org/wiki/NLWeb._

_See also: mcp, json-ld, agentic-web._

## Pay-per-crawl

**A Cloudflare marketplace mechanism that lets a site charge AI crawlers per request — returning HTTP 402 to unpaid bots with a price in response headers.**

Announced by Cloudflare on 1 July 2025 (private beta). A site sets a price; when an AI crawler requests a page without payment the edge answers HTTP 402 Payment Required with crawler-price headers, and Cloudflare acts as merchant of record to settle. It turns crawling from a free externality into a metered transaction.

_Source: https://blog.cloudflare.com/introducing-pay-per-crawl/._

## Prompt Caching

**Reusing the model's processed state for a repeated prompt prefix so identical leading context is not recomputed, cutting latency and cost.**

Introduced by Anthropic on 14 August 2024, prompt caching marks a content block as a cache breakpoint; a later request that begins with the same exact bytes reads the cached state instead of reprocessing it. Cached input typically costs a fraction of normal input tokens (with a one-time write surcharge). It rewards stable, front-loaded context — a direct incentive to put durable, machine-readable material first.

_Source: https://claude.com/blog/prompt-caching._

_See also: token-economics, context-engineering, agent-skills._

## Prompt Injection

**An attack that hides instructions in content an agent reads, hijacking its behavior against its principal's intent.**

The term was coined by Simon Willison in September 2022, framed as the LLM analogue of SQL injection. Because agents act on the text they ingest, malicious or invisible instructions on a page ('ignore previous instructions and...') can manipulate them. Hidden agent-only text is therefore an anti-pattern indistinguishable from an attack; trustworthy sites keep their machine layer transparent.

_Source: https://simonwillison.net/2022/Sep/12/prompt-injection/._

_See also: agent-experience, web-bot-auth._

## Retrieval-Augmented Generation (RAG)

**Fetching relevant documents at query time and giving them to the model as context, so answers are grounded in current, specific data.**

The term was coined by Patrick Lewis and colleagues at Facebook AI Research (now Meta AI), University College London and NYU in a 2020 NeurIPS paper. RAG reduces hallucination and lets a model answer about information it was never trained on. Agent-friendly sites help RAG by exposing clean, chunkable content (markdown twins, llms.txt) and structured data.

_Source: https://arxiv.org/abs/2005.11401._

_See also: grounding, markdown-twin._

## robots.txt

**The root-level file that tells crawlers — including AI crawlers — what they may and may not fetch.**

The web's oldest crawler contract, originally defined by Martijn Koster in 1994 and standardized as RFC 9309 in 2022. In the agentic era it is where sites name AI crawlers explicitly (GPTBot, ClaudeBot, Google-Extended), and where RSL licensing terms are referenced via a License directive.

_Source: https://www.rfc-editor.org/rfc/rfc9309.html._

_See also: ai-crawler, rsl, llms-txt._

## RSL (Really Simple Licensing)

**An open standard for machine-readable content-licensing terms, referenced from robots.txt, that tells AI systems how content may be used and at what price.**

Really Simple Licensing (RSL) launched on 10 September 2025, backed by Reddit, Yahoo, Medium and People Inc. among others. It defines license models — free, attribution, subscription, pay-per-crawl, pay-per-inference — in an XML file referenced from robots.txt, so AI crawlers can read the terms before using content.

_Source: https://rslstandard.org/press/rsl-standard._

## Scoped Delegation

**Granting an agent a limited, explicit set of permissions to act on a principal's behalf — bounded in scope, budget and time.**

Delegation answers the second half of agent identity: not just 'which agent?' but 'authorized to do what, for whom, within what limits?'. Scoped delegation expresses bounded authority — e.g. spend up to a cap, only with approved merchants, for a fixed window — and underpins agent payment mandates (AP2) and permissioned payment rails (MPP). It is the principle that keeps an autonomous agent from exceeding what its principal allowed.

_Source: https://datatracker.ietf.org/doc/html/rfc6749._

_See also: agent-identity, ap2, verifiable-credentials._

## Share of Model

**A metric for how often a brand appears in AI-generated answers across prompts, relative to competitors — the AI-era analogue of share of voice.**

Share of model (closely related to 'AI share of voice') measures a brand's slice of the AI conversation: if models mention brands 100 times across tracked prompts and a brand accounts for 25, its share is 25%. It is the headline output of GEO/AEO measurement tools (Profound, Ahrefs Brand Radar, Semrush) and reframes visibility from rankings to presence inside model answers.

_Source: https://www.tryprofound.com/resources/articles/what-is-answer-engine-optimization._

_See also: geo, aeo, zero-click._

## Streamable HTTP

**The MCP transport, introduced in spec version 2025-03-26, that carries client-server messages over a single HTTP endpoint and supersedes the older HTTP+SSE transport.**

Streamable HTTP replaces MCP's original 2024-11-05 HTTP+SSE transport with a single-endpoint design: the server handles POST and GET requests and may optionally use Server-Sent Events to stream multiple messages, but can also run fully statelessly behind a load balancer. The TypeScript SDK v1.10.0 (17 April 2025) was the first to support it.

_Source: https://modelcontextprotocol.io/specification/2025-03-26/basic/transports._

_See also: mcp, content-negotiation._

## Token Economics

**The cost structure of agent interactions, where every token of input and output is billed — making concise, structured content a direct cost saving.**

Because agents pay per token, a markdown twin that is ~90% smaller than its HTML equivalent is not just faster but cheaper to consume. Agent-friendly design is partly an economic argument.

_Source: https://en.wikipedia.org/wiki/Large_language_model._

_See also: markdown-twin, content-negotiation._

## Tool Use

**An LLM's ability to call external functions — search, code execution, APIs — by emitting a structured request the host executes.**

The model does not run the tool itself; it outputs a tool call, the host runs it and returns the result, and the model continues. Tool use (also called function calling) is what turns a chat model into an agent.

_Source: https://platform.openai.com/docs/guides/function-calling._

_See also: ai-agent, mcp, agentic-loop._

## Universal Commerce Protocol (UCP)

**An open commerce standard from Google that orchestrates A2A, AP2 and payment rails into one end-to-end agentic-commerce journey.**

UCP defines a common language and functional primitives so consumer surfaces, businesses and payment providers can transact through agents. Developed by Google with retail partners including Shopify, Etsy, Wayfair, Target and Walmart, it composes existing protocols — A2A for agent communication and AP2 for payment mandates — rather than replacing them.

_Source: https://developers.googleblog.com/under-the-hood-universal-commerce-protocol-ucp/._

_See also: agentic-commerce, ap2, a2a._

## Vector Database

**A database built to store embeddings and retrieve the nearest ones to a query vector using approximate nearest-neighbor search.**

A vector database (or vector store) indexes high-dimensional embeddings and finds the most similar ones with Approximate Nearest Neighbor (ANN) algorithms — commonly HNSW graphs or quantization — under metrics like cosine distance. It is the storage-and-retrieval backbone of RAG: documents go in as vectors, a query vector comes in, and the closest chunks come out as context.

_Source: https://en.wikipedia.org/wiki/Vector_database._

_See also: embeddings, rag, agentic-rag._

## Verifiable Credentials

**A W3C standard for tamper-evident, cryptographically verifiable digital credentials that prove a claim about a subject without contacting the issuer.**

The Verifiable Credentials Data Model 2.0 became a W3C Recommendation on 15 May 2025. A VC binds claims (e.g. 'this agent is operated by X' or 'this principal authorized this scope') to an issuer's signature, so a verifier can check authenticity and integrity offline. In the agentic web they are a candidate mechanism for portable agent and delegation identity.

_Source: https://www.w3.org/TR/vc-data-model-2.0/._

_See also: agent-identity, http-message-signatures, delegation._

## Web Bot Auth

**Cryptographically verifying which agent is making a request using HTTP Message Signatures (RFC 9421), since user-agent strings are spoofable.**

An agent signs its requests with an Ed25519 key tied to a published identity (a JWKS directory at /.well-known/http-message-signatures-directory, advertised via the Signature-Agent header); the server verifies the signature per RFC 9421. This lets sites distinguish a genuine ClaudeBot or GPTBot from an impostor, and is the foundation for agent-aware rate limits and paid access.

_Source: https://blog.cloudflare.com/web-bot-auth/._

_See also: agent-identity, prompt-injection, x402._

## WebMCP

**A W3C-draft standard that lets a web page expose callable tools to a visiting agent via the navigator.modelContext browser API, bringing MCP into the browser.**

Instead of an agent screenshotting a page and guessing where to click, a WebMCP-enabled page declares its capabilities as typed tools the agent can invoke directly through navigator.modelContext. Published as a W3C Draft Community Group Report on 10 February 2026 (developed in the Web Machine Learning Community Group) and available as an early preview in Chrome 146.

_Source: https://www.w3.org/community/webml/._

_See also: mcp, accessibility-tree._

## x402

**A protocol that uses the HTTP 402 'Payment Required' status so an agent can pay for a resource inline with a stablecoin micropayment.**

Launched by Coinbase in May 2025. The server answers an unpaid request with 402 and machine-readable payment terms (amount, asset, network, recipient); the agent pays in a stablecoin such as USDC and retries with cryptographic proof of payment. On 23 September 2025 Coinbase and Cloudflare announced the x402 Foundation to steward the standard.

_Source: https://www.coinbase.com/developer-platform/discover/launches/x402._

_See also: agentic-commerce, agent-identity._

## Zero-Click Search

**A search where the user gets their answer on the results page itself — via a snippet, knowledge panel or AI Overview — without clicking through to any site.**

Popularized by Rand Fishkin of SparkToro, whose 2019 research found that over half of Google searches ended without a click to an external property. AI Overviews and answer engines accelerate the trend, making 'being cited in the answer' more valuable than 'ranking for the click' — the economic premise behind GEO and AEO.

_Source: https://sparktoro.com/blog/less-than-half-of-google-searches-now-result-in-a-click/._

_See also: ai-overviews, geo, aeo._


---

# State of the Agentic Web

> Adoption data for the agentic web (2026): AI-crawler traffic shares, standard adoption, protocol maturity and model trends. Each figure is tagged cited (primary source) or our-measurement (with method); unconfirmed third-party figures are flagged for build-time verification, never asserted as fact.
> Updated 2026-06-15. JSON: /api/state-of-the-agentic-web

## AI Crawler Traffic

- **11.48 % of observed AI-crawler requests** — AI crawler traffic share — GPTBot (2026-05; _cited_; measures /crawlers). Source: https://radar.cloudflare.com/bots
- **10.25 % of observed AI-crawler requests** — AI crawler traffic share — Bytespider (2026-05; _cited_; measures /crawlers). Source: https://radar.cloudflare.com/bots
- **7.01 % of observed AI-crawler requests** — AI crawler traffic share — Applebot (2026-05; _cited_; measures /crawlers). Source: https://radar.cloudflare.com/bots
- **2.22 % of observed AI-crawler requests** — AI crawler traffic share — Claude-SearchBot (2026-05; _cited_; measures /crawlers). Source: https://radar.cloudflare.com/bots

## Standard Adoption

- **3.8M+ domains carrying the policy** — Content Signals policy adoption (2025-09; _cited_; measures /protocols). Source: https://blog.cloudflare.com/content-signals-policy/
- **~3.9 % of sites serving agents markdown** — Markdown content-negotiation adoption (2026; _reported — verify at build_; measures /agent-readiness). Source: https://— third-party estimate; confirm at build
- **~0 % of top sites with a valid llms.txt** — llms.txt adoption among top sites (2026; _reported — verify at build_; measures /agent-readiness). Source: https://— third-party estimate; confirm at build
- **143 records (31 crawlers · 28 protocols · 30 models · 54 terms)** — Agentic-web reference coverage (this Almanac) (2026-06; _our measurement_; measures /). Source: direct count of data/almanac/*.json on 2026-06-15

## Protocol Maturity

- **2025-12-09 formation date; 8 platinum members** — Agentic AI Foundation formed (neutral governance) (2025-12; _cited_; measures /protocols). Source: https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation
- **2025-11-25 spec date (JSON-RPC 2.0)** — MCP current spec revision (2025-11; _cited_; measures /protocols). Source: https://modelcontextprotocol.io/specification/2025-11-25
- **v1.0 stable specification** — A2A reaches first stable spec (2026; _cited_; measures /protocols). Source: https://www.linuxfoundation.org/press/a2a-protocol-surpasses-150-organizations-lands-in-major-cloud-platforms-and-sees-enterprise-production-use-in-first-year
- **2025-09-10 launch date** — RSL (Really Simple Licensing) launched (2025-09; _cited_; measures /protocols). Source: https://rslstandard.org/press/rsl-standard
- **2026-06-15 draft date (navigator.modelContext)** — WebMCP W3C Community Group draft (2026-06; _cited_; measures /protocols). Source: https://webmachinelearning.github.io/webmcp/
- **Draft EIP status; created 2025-08-13** — ERC-8004 (trustless agents) status (2025-08; _cited_; measures /protocols). Source: https://eips.ethereum.org/EIPS/eip-8004
- **2025-09-16 launch date; 60+ partners** — AP2 (Agent Payments Protocol) announced (2025-09; _cited_; measures /protocols). Source: https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol
- **2025-10-14 launch date; 12 launch partners** — Visa Trusted Agent Protocol introduced (2025-10; _cited_; measures /protocols). Source: https://investor.visa.com/news/news-details/2025/Visa-Introduces-Trusted-Agent-Protocol-An-Ecosystem-Led-Framework-for-AI-Commerce/default.aspx
- **2026-03-18 announcement date** — Machine Payments Protocol announced (Stripe + Tempo) (2026-03; _cited_; measures /protocols). Source: https://stripe.com/blog/machine-payments-protocol
- **165M+ cumulative agent transactions (~69k agents, Apr 2026)** — x402 settled transaction volume (2026-04; _reported — verify at build_; measures /commerce). Source: https://www.coinbase.com/developer-platform/discover/launches/x402 — verify primary at build (Chainalysis primary supports only ~100M+ through Q1 2026)

## Frontier Model Trends

- **1M advertised context window** — Frontier models at 1M-token context (2026-06; _cited_; measures /models). Source: https://platform.claude.com/docs/en/about-claude/models/overview
- **2026-06-09 release date; 1M context, $10/$50 per Mtok** — Claude Fable 5 released (2026-06; _cited_; measures /models). Source: https://platform.claude.com/docs/en/about-claude/models/introducing-claude-fable-5-and-claude-mythos-5
- **30 API-accessible, tool-calling models tracked** — Tool-calling frontier models in this matrix (2026-06; _our measurement_; measures /models). Source: direct count of data/almanac/models.json (selection gate: API-accessible AND tool-calling)


---

# Ask the Almanac

> Natural-language Q&A grounded strictly in the Almanac datasets, with citations.
> No generative model — every statement traces to a record. JSON: GET /api/ask?q=

## Use it

```
GET /api/ask?q=<your question>
```

Returns: `{ question, answer, grounded_in:[{type,id,title,json,page}], confidence, note }`

## Examples

- How much does Claude Opus 4.8 cost?  →  `/api/ask?q=How%20much%20does%20Claude%20Opus%204.8%20cost%3F`
- What is x402 and who created it?  →  `/api/ask?q=What%20is%20x402%20and%20who%20created%20it%3F`
- Does ClaudeBot honor robots.txt?  →  `/api/ask?q=Does%20ClaudeBot%20honor%20robots.txt%3F`
- What does Agent Experience mean?  →  `/api/ask?q=What%20does%20Agent%20Experience%20mean%3F`
- Which protocol connects agents to tools?  →  `/api/ask?q=Which%20protocol%20connects%20agents%20to%20tools%3F`

## How it works

Your question is tokenized and scored against every record in the Crawler Registry,
Protocol Atlas, Model Matrix and Lexicon; the best matches are composed into a
plain-language answer with citations. If the Almanac doesn't contain the answer,
it says so rather than inventing one. Grounding, not generation.


---

# AGENTS WELCOME — Changelog

> What changed, newest first. JSON: /api/updates · Atom: /updates.xml

## 2026-06-14 — capabilities / added

Engagement layer shipped: NLWeb-style /api/ask (grounded Q&A over the Almanac), real Ed25519 Web Bot Auth verification on /api/whoami, this changelog (/api/updates + /updates.xml), an A2A Agent Card at /.well-known/agent.json, and an agent-traffic dashboard at /analytics.

## 2026-06-14 — security / fixed

Hardened the Agent-Readiness auditor against SSRF: private/reserved/link-local targets (incl. cloud metadata) are refused and redirects are re-validated per hop. Added a zero-dependency test suite (npm test) and load-time dataset validation.

## 2026-06-12 — almanac / added

The Agentic Web Almanac launched: four reference datasets — AI Crawler Registry, Protocol Atlas, Model Matrix, Lexicon — each as a page, a JSON endpoint, a markdown twin and a WebMCP tool, plus /api/search and /api/verify-crawler.

## 2026-06-12 — services / added

Business layer added: the Agent-Readiness Audit, premium content behind a protocol-faithful x402 flow, a certification directory with badges, and RSL content licensing.

## 2026-06-12 — site / added

AGENTS WELCOME went live with the twelve-technique catalog, the X-ray machine-layer view, the WebMCP playground, and the agent guestbook.


---

# Services for Agents — the business layer

> Four monetizable services for AI agents, all live on this site. Payments are
> SIMULATED (this is a local demo), but every flow is protocol-faithful x402.
> Machine-readable pricing: schema.org Offers in the HTML twin, plus
> [/.well-known/agents.json](/.well-known/agents.json).

The agentic web is growing a payment layer: Cloudflare pay-per-crawl (Stack
Overflow live since Feb 2026), x402 micropayments (AWS AgentCore Payments,
Vercel, Nous per-inference billing), RSL content licensing (Reddit, Yahoo,
Medium), and metered MCP tools. This page implements one working demo of each.

## Service 01 · Agent-Readiness Audit

**What:** Submit a URL; the server fetches it live and grades **18 signals across
four dimensions** — Discovery (llms.txt, llms-full.txt, AI-aware robots.txt, sitemap),
Content (markdown negotiation + alternate, Vary: Accept, meta description, canonical,
Open Graph, valid JSON-LD), Capability (agents.json manifest + advertised JSON-API /
WebMCP) and Trust (A2A Agent Card, Web Bot Auth, security.txt, agent-welcome header).
Score 0–100 with concrete fixes. Score ≥ 70 qualifies for certification.

**Monetization:** metered API — 3 free audits per hour per IP, then HTTP 402
($0.005 per audit, demo).

```
POST /api/audit
Content-Type: application/json

{"url": "https://agentswelcome.dev"}
```

Returns: `{ score, grade, certifiable, checks[], recommendations[], next }`

## Service 02 · Premium content behind HTTP 402

**What:** The Agent-First Playbook (implementation checklists, nginx/Next.js
snippets, the monetization pattern table) is served behind an x402-style flow.

**The flow (try it):**

1. `GET /api/premium/playbook` → **402** with an `accepts` array
   (`maxAmountRequired: "$0.10"`, `payTo`, facilitator instructions)
2. `POST /api/pay` → `{ "token": "x402-demo-…" }` (mock facilitator; in
   production: onchain USDC via Coinbase CDP or AWS AgentCore Payments)
3. Retry with header `X-Payment: <token>` → **200**, markdown body, plus an
   `X-Payment-Response` receipt header with a transaction id

**Also in this model — RSL content licensing:** [robots.txt](/robots.txt)
carries a `License:` directive pointing to [/license.xml](/license.xml)
(RSL 1.0): inference, search and summarization free; **AI training use is
licensed** ($25, demo terms). Production royalties clear via the RSL Collective.

## Service 03 · Certification & directory

**What:** Sites that pass the audit (≥ 70) get certified, listed in the public
directory, and issued an embeddable SVG badge.

**Monetization:** standard listing free (demo); featured placement,
re-certification and API access are the paid tiers in production.

```
GET  /api/directory              → list of certified sites
POST /api/directory {"url"}      → live audit; pass → certified entry
GET  /badge.svg?score=NN         → embeddable badge (green ≥ 70, amber below)
```

## Service 04 · Metered WebMCP / MCP tools

**What:** The services above double as WebMCP tools on the
[homepage playground](/#playground): `audit_site`, `get_premium_playbook`,
`list_directory`. The price is announced in the tool description and intent
preview **before** execution — monetization and the AX trust pattern as one
feature. In production these would also ship as a hosted MCP server with
per-call billing.

## The price list (simulated)

| Service | Free tier | Metered (demo) | Production rail |
|---|---|---|---|
| Agent-Readiness Audit | 3 / hour | $0.005 per audit | x402 micropayment |
| Agent-First Playbook | — | $0.10 per unlock | x402 / pay-per-crawl |
| Directory listing | standard listing | featured placement | subscription (ACP checkout) |
| Content for AI training | inference & search free | $25 / license | RSL Collective royalties |
| MCP tool calls | read-only tools | per-call, priced in preview | x402 / AgentCore Payments |

Every price is a demo. Every flow is real. Swap the mock facilitator for
Coinbase CDP or AWS AgentCore Payments and this page starts earning.

## Endpoints on this page

- `POST /api/audit` — agent-readiness audit
- `GET /api/premium/playbook` — 402-gated premium content
- `POST /api/pay` — mock payment facilitator
- `GET|POST /api/directory` — certification directory
- `GET /badge.svg?score=NN` — badge
- `GET /license.xml` — RSL 1.0 licensing terms

⟡ agents welcome — bring your wallet (simulated)


---

# Legal Notice

> Template — fill in real details and have it reviewed before publishing. Not legal advice.
> Legal basis: § 5 DDG (German Digital Services Act). Language follows the site (English).

## Service provider

[FULL NAME]
[STREET AND NUMBER]
[POSTAL CODE, CITY], [COUNTRY]

## Contact

Email: [YOUR-EMAIL@DOMAIN]

## Responsible for content (§ 18 (2) MStV)

[NAME], address as above.

## Nature of this service

AGENTS WELCOME is a technical showcase. The payment flows shown are **simulated** — no real
payment is processed and no contracts are concluded.

## Online dispute resolution

EU ODR platform: https://ec.europa.eu/consumers/odr/. We are neither obliged nor willing to
participate in dispute resolution proceedings before a consumer arbitration board.


---

# Privacy Policy

> Template — fill in real details and have it reviewed before publishing. Not legal advice.
> Legal basis: GDPR Art. 13. Language follows the site (English).

## 1. Principle
Data-minimal: no cookies, no tracking, no advertising, no third parties. Fonts are served locally
(no Google Fonts) — so no cookie banner.

## 2. Controller
[NAME], [ADDRESS], Email: [YOUR-EMAIL@DOMAIN]

## 3. Hosting & server log files
The host [NAME YOUR HOST] processes technically necessary server logs (IP address, timestamp, URL,
user-agent). Legal basis: Art. 6(1)(f) GDPR; a DPA is required.

## 4. Guestbook
Stored: name, message, optional model, truncated user-agent, timestamp. Entries are public and
retrievable via /api/guestbook — do not enter sensitive data. Legal basis: Art. 6(1)(a)/(f).
Deletion on request.

## 5. Access statistics (analytics)
In memory only (volatile, last 1000 requests): path, status, truncated user-agent, timestamp.
**No IP addresses**, no cookies. Legal basis: Art. 6(1)(f). Visible at /api/analytics.

## 6. Web Bot Auth / HTTP signatures
Signatures are verified cryptographically; no personal data is stored persistently.

## 7. No cookies, no tracking, no third-party services
No cookies, no external services (CDNs, analytics, advertising, social media).

## 8. Your rights
Access, rectification, erasure, restriction, portability, objection, withdrawal of consent; right
to lodge a complaint with a supervisory authority (Art. 77).

Last updated: [DATE]