AI Crawler Index

AI Crawler Index is a free, live data page published by the digital marketing agency Mojo Dojo. It logs every crawler that visits mojodojo.io and classifies each one as an AI training bot, AI search bot, AI user fetch, search engine, SEO tool, social preview bot, archive or other bot. Mojo Dojo states that AI crawlers made 26% of its bot traffic over the last 30 days.

What It Is

The Index is a case-study dashboard of crawler activity on a single website, not a tool you install. It reports hits, hits per day, distinct pages visited, hits per page, verification rate and fake hits caught for each bot, with 7, 30 and 90 day views. Mojo Dojo notes that the figures come from one site and should be read as a case study rather than a census.

How the Numbers Are Made

Every page request whose user agent looks like a crawler is logged at the web server together with the visitor's IP. A hit counts as verified when the IP falls inside ranges published by the operator, or passes forward-confirmed reverse DNS. Hits that fail the check are marked spoofed and excluded from all shares. Bots whose operators publish no verification method are counted as unverifiable. IP recording began on 7 Oct 2026, so earlier hits are classified by user agent only.

What the Dashboard Shows

• Daily crawler hit trend with spoofed traffic removed.
• Category split showing the share of hits per bot type.
• A ranked leaderboard of bots covering GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot and others.
• Bot awards such as most aggressive, most polite, most thorough and most impersonated.

The data is free to reuse under CC BY 4.0 with a link back to the page.