# agent-traffic

> Open-source dataset and classifier that tells AI agents, AI crawlers, search crawlers and bots apart from human HTTP traffic.

agent-traffic is an MIT-licensed project from Mintlify that classifies HTTP requests. It combines a curated, tested dataset of user agents, headers and official verification sources for real agents and crawlers with a small `classify` function that applies it. It ships as the npm package `@mintlify/agent-traffic`.

## What It Is

agent-traffic is a bot and agent detection library. Given a user agent, request headers and optionally a client IP, it returns a category (`agent`, `ai-crawler`, `crawler`, `bot`, `human` or `unknown`), a more specific kind, the product name and operator where known, a verification flag, a confidence level and a list of reasons explaining the result.

The project's stated motivation is that simple bot lists match user agent substrings, accept whatever a request claims, and lump all automation together. Mintlify says it built this after studying traffic across the documentation sites it hosts.

## How classification works

Checks run in order and the first to answer wins: known agent or crawler (verified by IP where possible), a request asking for markdown, generic HTTP client or headless browser, a browser unless its headers contradict it, and otherwise a bot.

- Crawlers can be checked against IP ranges published by operators such as OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft, Common Crawl and DuckDuckGo, so a spoofed GPTBot is flagged as an impersonator.
- Agents that send a browser user agent but prefer `text/markdown` in the Accept header are identified as unidentified agents.
- Optional fake-browser checks are enabled with `headersComplete`.
- Coding agents that run on a user's own machine are never IP-checked.

## Using it

The package has zero dependencies and runs wherever `fetch` does, including Node, Bun, Deno, Cloudflare Workers and Vercel Edge. A typical use is a request handler that serves markdown to agents. It also supports classifying stored logs, with patterns translatable to ClickHouse, DuckDuckGo-style DuckDB and Postgres queries, and exposes per-category regular expressions plus the dataset as JSON.

An installable skill (`npx skills add mintlify/agent-traffic`) teaches coding agents such as Claude Code, Cursor and Codex how to use the package.

## Known limits

Some products share an identity, such as Codex reportedly fetching as ChatGPT-User. Agentic browsers running inside a person's own Chrome look like that person at the HTTP level. Logs that keep only the user agent will miss agents using browser user agents unless the Accept header is also logged.

## Features
- Request classification into agent, ai-crawler, crawler, bot, human and unknown categories
- Curated dataset of agent and crawler user agents, headers and verification sources
- IP range verification against operators' published ranges
- Impersonator detection
- Markdown Accept-header detection for agents using browser user agents
- Fake-browser header checks
- Human-readable reasons and confidence for every result
- Per-category regex patterns
- Zero dependencies; runs on Node, Bun, Deno, Cloudflare Workers and Vercel Edge
- Installable agent skill for coding agents

## Integrations
Claude Code, Cursor, Codex, Cloudflare Workers, Vercel Edge, ClickHouse, DuckDB, Postgres

## Platforms
WEB, API, VSC_EXTENSION, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Links
- Website: https://agent-traffic.mintlify.site
- Documentation: https://agent-traffic.mintlify.site
- Repository: https://github.com/mintlify/agent-traffic
- EveryDev.ai: https://www.everydev.ai/tools/agent-traffic
