agent-traffic
Open-source dataset and classifier that tells AI agents, AI crawlers, search crawlers and bots apart from human HTTP traffic.
At a Glance
The agent-traffic package and dataset are released under the MIT License and are free to use, copy, modify and distribute.
Engagement
Available On
Alternatives
Listed Nov 2026
About agent-traffic
agent-traffic is an MIT-licensed project from Mintlify that classifies HTTP requests. It combines a curated, tested dataset of user agents, headers and official verification sources for real agents and crawlers with a small classify function that applies it. It ships as the npm package @mintlify/agent-traffic.
What It Is
agent-traffic is a bot and agent detection library. Given a user agent, request headers and optionally a client IP, it returns a category (agent, ai-crawler, crawler, bot, human or unknown), a more specific kind, the product name and operator where known, a verification flag, a confidence level and a list of reasons explaining the result.
The project's stated motivation is that simple bot lists match user agent substrings, accept whatever a request claims, and lump all automation together. Mintlify says it built this after studying traffic across the documentation sites it hosts.
How classification works
Checks run in order and the first to answer wins: known agent or crawler (verified by IP where possible), a request asking for markdown, generic HTTP client or headless browser, a browser unless its headers contradict it, and otherwise a bot.
- Crawlers can be checked against IP ranges published by operators such as OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft, Common Crawl and DuckDuckGo, so a spoofed GPTBot is flagged as an impersonator.
- Agents that send a browser user agent but prefer
text/markdownin the Accept header are identified as unidentified agents. - Optional fake-browser checks are enabled with
headersComplete. - Coding agents that run on a user's own machine are never IP-checked.
Using it
The package has zero dependencies and runs wherever fetch does, including Node, Bun, Deno, Cloudflare Workers and Vercel Edge. A typical use is a request handler that serves markdown to agents. It also supports classifying stored logs, with patterns translatable to ClickHouse, DuckDuckGo-style DuckDB and Postgres queries, and exposes per-category regular expressions plus the dataset as JSON.
An installable skill (npx skills add mintlify/agent-traffic) teaches coding agents such as Claude Code, Cursor and Codex how to use the package.
Known limits
Some products share an identity, such as Codex reportedly fetching as ChatGPT-User. Agentic browsers running inside a person's own Chrome look like that person at the HTTP level. Logs that keep only the user agent will miss agents using browser user agents unless the Accept header is also logged.
Community Discussions
Be the first to start a conversation about agent-traffic
Share your experience with agent-traffic, ask questions, or help others learn from your insights.
Pricing
Open Source (MIT)
The agent-traffic package and dataset are released under the MIT License and are free to use, copy, modify and distribute.
- MIT-licensed TypeScript package @mintlify/agent-traffic
- Curated dataset of user agents, headers and verification sources
- classify() function with IP range verification
- Zero dependencies; runs on Node, Bun, Deno, Cloudflare Workers, Vercel Edge
- Install via npm i @mintlify/agent-traffic
Capabilities
Key Features
- Request classification into agent, ai-crawler, crawler, bot, human and unknown categories
- Curated dataset of agent and crawler user agents, headers and verification sources
- IP range verification against operators' published ranges
- Impersonator detection
- Markdown Accept-header detection for agents using browser user agents
- Fake-browser header checks
- Human-readable reasons and confidence for every result
- Per-category regex patterns
- Zero dependencies; runs on Node, Bun, Deno, Cloudflare Workers and Vercel Edge
- Installable agent skill for coding agents
