The Agentic-Web Tooling Landscape

What is an agentic-web tool?

An agentic-web tool is a product that measures, analyzes, tracks, or audits how a website participates in the agentic web. This landscape reviews eight tools across four jobs — adoption measurement, crawler analytics, AI-visibility tracking and readiness checking — with the same fields for each: what it does, strength, weakness, monetization and when to use it. Every price, plan name and count was checked against the vendor’s own site on 7 September 2026 (URLs in the Sources list); where a vendor publishes no number — Known Agents shows no bot total — this page says so instead of estimating.

A tool is not a standard and not a service: a tool measures or checks something (Cloudflare Radar, Known Agents), a standard specifies a file or protocol that tools read and write (llms.txt, Web Bot Auth), and a service engineers and certifies the result (the Agent-Readiness Audit).

Which four categories organize the agentic-web tooling landscape?

Agentic-web tools fall into four categories — adoption measurement, crawler analytics, GEO / AI-visibility and readiness checking — and each category feeds a different Almanac pillar, as the table shows.

What does Cloudflare offer: the Agent Readiness score, Radar data and pay-per-crawl?

Cloudflare is the broadest agentic-web tool: since 17 April 2026 it has published a free Agent Readiness score with a public scanner at isitagentready.com, its Radar service reports network-wide crawl traffic by bot, and its AI Crawl Control product — formerly AI Audit, available on all plans — lets a site see which AI services access its content, control access with granular policies and, in a closed beta, charge AI crawlers (Cloudflare, 2026).

• What it does: scores four dimensions — Discoverability, Content, Bot Access Control and Capabilities — and, through pay-per-crawl, answers an unpaid crawler with HTTP 402 and requires it to sign requests with HTTP Message Signatures, the mechanism behind Web Bot Auth (Cloudflare, July 2025).
• Strength: grounded in a large sample — Cloudflare scanned the 200,000 most visited domains and found a robots.txt on 78 % of them, declared AI-usage preferences on 4 %, markdown content negotiation on 3.9 %, and MCP or API-catalog standards on fewer than 15 sites (Cloudflare, April 2026). Radar’s 2025 Year in Review put GPTBot at about 7.5 % of verified-bot traffic and all AI bots at 4.2 % of HTML requests on average (Cloudflare, December 2025); this Almanac’s State of the Agentic Web dataset records Radar’s May 2026 AI-crawler shares — GPTBot 11.48 %, Bytespider 10.25 %, Applebot 7.01 %, Claude-SearchBot 2.22 % — with source and last-verified date (2026-06-15).
• Weakness: ecosystem-locked — enforcement and pay-per-crawl only work for sites on Cloudflare; Cloudflare-flavored framing; no teaching or reference layer for what the numbers mean.
• Monetization: free scanner and Radar data; AI Crawl Control on all plans; pay-per-crawl in closed beta with Cloudflare as merchant of record (Cloudflare, 2025).
• Our take: the best free first scan and the canonical adoption data source. Cloudflare’s pay-per-crawl prices AI access at the edge, and its verification relies on Web Bot Auth, which verifies which crawler the analytics tools count.

What does Known Agents (formerly Dark Visitors) do?

Known Agents — the crawler-analytics service formerly named Dark Visitors — shows a site every crawler, scraper and AI agent visiting it in real time, measures human referrals from ChatGPT, Claude, Gemini and other AI chat platforms, serves a robots.txt that updates automatically as new bots are discovered, identifies agents from incoming requests through an API, and lists agents in a public directory filtered by 17 categories — the deepest crawler list in this landscape. It publishes no total bot count, and its free plan covers one project (Known Agents, verified 7 September 2026).

• Strength: breadth and detail — a page per agent across 36 directory pages (7 September 2026); the community block-list ai.robots.txt credits Known Agents as a source for its crawler entries (ai.robots.txt README, read 2026-09-07); an official Node.js SDK is open source on GitHub under Apache-2.0 (Known Agents, node-sdk).
• Weakness: crawler-only — no protocols, models or readiness reference; any bot total you read elsewhere is reported, not audited; the Agent Identification API and CSV export start at the paid Startup tier.
• Monetization: freemium — Free ($0: one project, 100K events, automatic robots.txt), Startup ($29 per month: three projects, 1M events), Growth ($299 per month: ten projects, 10M events) and Enterprise (custom), as listed on 7 September 2026.
• Our take: the tool to identify and allow or block specific crawlers; our registry organizes the AI crawlers the analytics tools track in your logs neutrally and agent-first.

How do Profound, Ahrefs Brand Radar and Semrush track GEO visibility?

Profound, Ahrefs Brand Radar and Semrush are GEO / AI-visibility tools that track whether and how often AI answer engines mention or cite your brand — they measure the outcome (visibility) but do not engineer the cause (the readiness signals that earn citations). As of 7 September 2026, entry prices start at $99 per month (Profound Starter, billed yearly; Semrush AI Visibility Toolkit, billed annually), and Ahrefs includes Custom Prompts in every paid plan. The table names what each one tracks, what it costs to start, and who it fits.

• Strength: quantifies citation presence across engines, turning “are we cited?” into a tracked metric — and the metric moves: in Profound’s study of about 80,000 prompts per platform, 40–60 % of the domains cited for the same prompts had changed one month later (June to July 2025: Google AI Overviews 59.3 %, ChatGPT 54.1 %, Microsoft Copilot 53.4 %, Perplexity 40.5 %; Profound, July 2025).
• Weakness: measurement-only — they tell you whether you are cited, not how to get cited; this is the GEO measurement-vs-engineering split. Prices scale with tracked prompts and engines.
• Monetization: SaaS subscriptions (entry prices in the table).
• Our take: use these to measure AI visibility; use the GEO engineering guide and the Audit to change it. The trackers measure what GEO (Generative Engine Optimization) defines.

What does an llms.txt generator do?

An llms.txt generator writes the /llms.txt file that Jeremy Howard proposed on 3 September 2024: a markdown file at a site’s root with brief background information, guidance and links to detailed markdown files (llmstxt.org, 2024) — and, where the generator supports it, a full-text companion such as the /llms-full.txt this site serves. It is the cheapest first step toward agent readiness — a starting point, not a finished posture.

• Strength: near-zero effort and immediate machine-readability; the specification itself is free and open.
• Weakness: single-signal — it covers only the declaration layer, validates nothing else about the site, and output quality varies by generator. Cloudflare’s Agent Readiness scan does not check llms.txt by default; the scan can be customized to include it (Cloudflare, April 2026).
• Monetization: varies by generator; the standard itself is free.
• Our take: generate the file, then verify the whole readiness posture — llms.txt is only Layer-1 discovery: a generator writes the file the llms.txt standard specifies, then you implement llms.txt after a generator drafts it.

What do AgentReady and specification.website specify?

AgentReady is an open, MIT-licensed, versioned standard for agent readiness that Ora published on 20 March 2026 (v1.0.0 on 24 April 2026), with stable requirement identifiers such as AR-DISC-01 and AR-CAPA-01 across five sections — Discoverability, Content for agents, Capabilities, Identity & Access, Commerce — and a free reference scanner, Deep Scan, that returns a 0–100 score with a letter grade (Ora, 2026). specification.website is Joost de Valk’s “platform-agnostic, full specification of the technical features a good website should have”: MIT-licensed code, CC BY 4.0 content, a source cited on every page, and agent readiness as one of its ten domains (de Valk, 2026).

• What it does: AgentReady publishes the requirements a site must meet so an agent can find, read and act — as a page and as spec.json at agentready.org — and scores sites through Deep Scan (Ora, 2026).
• Strength: open, citable by identifier, and free to run.
• Weakness: a standard plus a scanner rather than a reference or glossary; its output is a score and a letter grade.
• Monetization: standard and scanner free; published by a vendor, Ora.
• Our take: cite it as a neutral peer and the closest public overlap with our Audit; its five sections map almost one-to-one onto the six readiness dimensions this site audits.

• What it does: specification.website collects the technical features a good website should have — HTML foundations, SEO, accessibility, security, well-known URIs, agent readiness, performance, privacy, resilience and internationalisation (de Valk, 2026).
• Strength: strong E-E-A-T (authored by Joost de Valk, the Yoast founder), open licences and a source on every page.
• Weakness: it specifies, it does not scan — it tells you what a site should have, not whether yours has it.
• Monetization: open source (MIT code, CC BY 4.0 content).
• Our take: cite it as a neutral peer; neither a specification nor a scanner issues the validated audit and certification badge this site does.

In-depth tool reviews

Each core tool has a dedicated, neutral review:

• Cloudflare AI Crawl Control — see, allow, block and charge AI crawlers at the edge.
• Known Agents (formerly Dark Visitors) — AI-agent analytics and robots.txt generation.
• Profound — how answer engines represent your brand (GEO visibility).
• Otterly.AI — brand-mention and citation monitoring across AI search.
• Firecrawl — turn the live web into LLM-ready markdown for agents.

Which tools does agentswelcome.dev itself expose to agents?

agentswelcome.dev exposes 12 WebMCP tools of its own, listed straight from data/tools.json — the file behind /api/tool-registry and the interfaces.webmcp.tools block of /.well-known/agents.json — so the list cannot drift. 10 of the 12 carry the WebMCP read-only annotation, 2 (sign_guestbook, get_premium_playbook) write or unlock something, and 2 run the x402 payment demo (demo — no real money). Every tool has an HTTP twin, so the same result needs no JavaScript.

Which agentic-web tool should you use for which job?

Each tool does one job and none certifies; the matrix puts all eight on the same seven columns, and a validating audit closes the loop.

The terms in this matrix are defined in the Lexicon that defines the agentic-web terms these tools use, and every standard a tool touches in the Protocol Atlas that defines the standards these tools touch.

Where does the Agent-Readiness Audit sit among these tools?

Among these tools, the Agent-Readiness Audit is a validating readiness check that — unlike a one-off scanner, a crawler list, or a visibility tracker — ties its result to a vendor-neutral reference and an “Agents Welcome” certification badge, the cleanest fully-unowned lane in the category. It does not claim to win on any single measurement axis; it adds whole-site validation and a certification none of the eight tools issues.

To confirm and certify what a free scanner flags, the Agent-Readiness Audit checks what the scanners flag — then certifies the result. And this is self-demonstrating: this site scores 100 on the readiness the scanners check — verify it live. The agents these tools track run on models scored in the Frontier Model Matrix that scores the models behind the agents.

Agentic-web tools — frequently asked questions

What is an agentic-web tool?

An agentic-web tool is a product that measures, analyzes, tracks, or audits how a website participates in the agentic web. They fall into four jobs: adoption measurement, crawler analytics, AI-visibility tracking, and readiness checking.

What is the best agent-readiness tool?

There is no single best tool — each one does a different job. Cloudflare's scanner is the best free first scan, Known Agents is the deepest crawler list, Profound, Ahrefs Brand Radar and Semrush measure AI visibility, and llms.txt generators are the cheapest first step. A validating audit then confirms and certifies the result, which none of the measurement or checklist tools does.

How do Profound, Ahrefs Brand Radar and Semrush differ?

All three are GEO / AI-visibility tools that track whether AI answer engines cite your brand. Profound is purpose-built for AI-answer monitoring, Ahrefs Brand Radar bolts AI-mention tracking onto an SEO suite, and Semrush offers an AI-visibility module inside a broad marketing platform. All measure the outcome; none engineers the readiness signals that earn citations.

Is an llms.txt generator enough to make a site agent-ready?

No. An llms.txt generator produces a content-declaration file — a fast, cheap first step — but it is a single signal, not a finished readiness posture. It does not validate that the rest of the site is machine-readable, so the file is a starting point, not a destination.

Where does Agents Welcome's own Audit fit among these tools?

The Agent-Readiness Audit is a validating readiness check that ties its result to a vendor-neutral reference and an 'Agents Welcome' certification badge. It does not beat any single tool on its own axis — Cloudflare is broader on edge data, Known Agents deeper on bots, the GEO trackers stronger on visibility — but it occupies the neutral-reference-plus-certification lane no other tool fully owns.

How do you get from measuring the agentic web to making your site ready for it?

Every tool above tells you where you stand; turning that reading into shipped changes — and a certified result — is the job of the Agent-Readiness Engineering guides that turn each scanner finding into a step and of the Audit.

This catalog feeds every pillar its tools touch: the Agent-Readiness Audit checks what the scanners flag, the adoption data these tools report is compiled over time, the crawler data Known Agents sells is organized agent-first, and the GEO guide engineers the citations these trackers measure. This neutral tooling review is one of the guides the agentic web home indexes for agents.