The Crawler Zoo

The Crawler Zoo is a live website that examines every request reaching it, identifies bots by user agent, and files them into themed enclosures while counting human visitors only as an anonymous daily total. Each bot gets a specimen page with a generated illustration, its operator, and guidance on how to block it. The site also offers a robots.txt generator and an AI-crawler blocking guide for site owners.

What It Is

The Crawler Zoo is a bot-traffic observatory presented as a zoo. Requests are checked against a catalogue of known crawlers, and large crawlers such as Googlebot, Bingbot and GPTBot are verified by reverse DNS or operator-published IP ranges. Impostors are filed in the Trap Room, and unrecognised automata go to the Feral Pit. Bots are grouped into five enclosures: Search Engine Savanna, The AI Aviary, SEO Swamp, Link-Preview Petting Zoo and The Feral Pit.

Specimen Files and Leaderboards

Every bot has a page showing what it is, who runs it, its exact user agent, and how to block it with a robots.txt rule or a server rule for bots that ignore robots.txt. Pages also show 30-day visits, most-requested pages, favourite hours, robots.txt record, and originating networks and countries. A leaderboard ranks species by requests for today, this week or all time, and stamps each with how it treats robots.txt.

Traps, Labyrinth and Arena

The robots.txt disallows /trap/, and every page hides a link to it that people cannot see. Bots that enter are listed in the Trap Room. Behind it, a generated endless maze called the Labyrinth lets visitors follow a bot live room by room. The Arena lets people enter a script or AI agent in five games, with boards reset weekly. A Feed the bots page lets visitors leave text snacks in per-enclosure files that crawlers can find.

Tools for Site Owners

The site provides a robots.txt generator with presets such as blocking AI training, a guide to blocking AI crawlers, and an embeddable badge showing a live visitor counter without scripts or cookies. Weekly Zoo Reports and an RSS feed summarise visitors, new species, impostors and trap catches. Per its documentation, bot addresses are never stored.