Known Agents reviewed: AI-agent analytics and robots.txt, formerly Dark Visitors

Known Agents (formerly Dark Visitors) tracks AI agent and bot traffic to your site in real time, generates a continuously-updated robots.txt to block misbehaving scrapers, and offers an agent-identification API plus a public agent directory.

What does Known Agents do?

Known Agents (formerly Dark Visitors) gives a site owner four products: real-time agent analytics that show every crawler, scraper and AI agent visiting the site; AI chat referral tracking for visits sent by ChatGPT, Claude, Gemini, Copilot, Perplexity, Meta AI, DeepSeek and Mistral; an automatic robots.txt that updates as new bots are discovered; and an Agent Identification API for verifying incoming requests (Known Agents, 2026). It also runs a public agent directory, the Agentic Web Index dashboard and a free AI Bot Robots.txt Checker, and states that more than 5,000 websites use it.

Where does Known Agents fit in the agentic web?

Known Agents is a crawler-analytics tool — which specific bots fetch my pages, and should I allow or block them? — serving the access-control dimension of agent readiness. Its catalogue overlaps the entities in the AI Crawler & Agent Registry, and its identification work is the job that crawler verification and Web Bot Auth address. The Registry is this Almanac's counterpart: 41 crawler records (38 active, 3 deprecated), each with its robots.txt token, or a note that it has none, and how to verify it, served live at /api/crawlers/{id}; /api/verify-crawler?ua= names the crawler a User-Agent string claims to be, because a User-Agent is a hint, not proof.

Our take

The specialist for identifying and gating individual crawlers; its live agent directory is genuinely useful when you write robots.txt rules. Like all analytics tools it measures and lists traffic; it does not certify that your site is agent-ready. Use its directory to know who is visiting, then the Agent-Readiness Engineering guide to decide what to expose to them.

How is the Known Agents agent directory organized?

The directory files agents into 17 categories — AI Agents, AI Assistants, AI Coding Agents, AI Data Providers, AI Data Scrapers, AI Search Crawlers, Archivers, Automated Agents, Developer Helpers, Fetchers, Intelligence Gatherers, Scrapers, Search Engine Crawlers, Security Scanners, SEO Crawlers, Uncategorized Agents and Undocumented AI Agents — across 36 pages, and it shows no total agent count (Known Agents, verified 7 September 2026). Sample entries that day: ChatGPT-User (AI Assistant), Claude-Code (AI Coding Agent), Bravebot (AI Data Provider). Any “N bots tracked” figure quoted elsewhere is therefore reported, not vendor-published.

Known Agents — frequently asked questions

What was Known Agents called before?

Dark Visitors. The vendor's rename announcement says people heard dark as bad rather than hidden, and states that existing integrations keep working with no updates required; the post itself carries no date.

Is the Known Agents automatic robots.txt free, and what do the paid plans cost?

Yes: the free plan ($0 per month, one project, 100K events) includes the automatic robots.txt. The paid plans are Startup at $29 per month (three projects, 1M events, Agent Identification API, CSV export, WordPress rule enforcement), Growth at $299 per month (ten projects, 10M events) and Enterprise at custom pricing (Known Agents, verified 7 September 2026).

How many agents does Known Agents track?

The vendor publishes no total. Its directory spans 36 pages in 17 categories and shows no count, so any figure quoted elsewhere is reported, not vendor-published.

Does Known Agents tell you whether your site is agent-ready?

No. It measures and lists agent traffic and lets you allow or block agents; readiness, meaning whether an agent can find, read and act on your content, is a separate check.

How do you install the Known Agents automatic robots.txt?

Through a WordPress, Cloudflare, Fastly or AWS connection, or from backend code via Node.js or the REST API (Known Agents docs, 2026); the official Node.js SDK is open source under Apache-2.0.

← Tooling Landscape · .md