crawlie

crawlie is a free, MIT-licensed technical SEO and Generative Engine Optimization (GEO) crawler built by Sean Ryan of Spronta Ltd. It runs locally as a tiny async Rust binary, requires no account or cloud connection, and ships both a CLI and a Model Context Protocol (MCP) server so AI agents can drive audits end-to-end. The project reached v0.4.0 in June 2026 and has 54 GitHub stars.

What It Is

crawlie is a command-line site auditor that crawls any website and runs 46 checks across two scoring dimensions: a Health score for classic technical SEO (broken links, redirects, metadata, canonicals, robots, performance, security, mobile/international signals) and a GEO score for AI-search readiness (structured data, semantic HTML, answer-ready content, authorship/E-E-A-T, dated content). Every finding includes plain-English guidance explaining why it matters, how to fix it, and what happens if it's ignored — not just an error code. The tool is written in Rust, targets wasm32 for future cloud-worker deployment, and is MIT licensed.

Agent-Native Architecture

crawlie's standout design choice is its first-class MCP server (crawlie-mcp). Any MCP-compatible agent — Claude, Cursor, Cline, or a custom agent — can call tools like crawl_site, audit_url, audit_urls, explain_issue, list_rules, and list_reports over JSON-RPC via stdio. A one-step Claude Code plugin bundles the MCP server and a set of pre-built audit skills (full-site SEO + GEO, broken-link fixes, pre-launch gates, AI-search readiness) that run on demand via npx without requiring a pre-installed binary. The skills folder also works with any agent even without the MCP server.

What It Checks

The 46 rules span four categories:

• Technical SEO: broken links, 4xx/5xx, redirect chains, titles and meta descriptions (missing/duplicate/length), H1s, canonicals, noindex/nofollow/X-Robots-Tag, robots.txt blocking, missing alt text, thin and duplicate content, orphan and deep pages
• Performance & security: slow responses, large pages, missing compression, HTTPS, mixed content, HSTS
• Mobile, international & social: viewport, lang, hreflang, Open Graph, Twitter cards, structured data
• GEO: structured data, semantic HTML, answer-readiness, authorship/E-E-A-T, dated content, question-style headings, extractable blocks

Setup Path and Output Formats

Installation is a single npm command (npm i -g crawlie), which automatically installs the correct native Rust binary as a platform package. A signed, notarized macOS .dmg desktop app (Tauri v2 + React) is available from GitHub Releases. Building from source requires Rust and, for the desktop app, pnpm and Node. Output formats include pretty (terminal), json (machine-readable default), csv (issues only), and html (self-contained shareable report). CI/CD gating is supported via --fail-on error|warning for non-zero exit codes.

Architecture and Codebase

The repository is organized into a crates/ directory with three crates — crawlie-core (engine, audit, scoring, knowledge base, reports), crawlie-cli (the crawlie command), and crawlie-mcp (the MCP server) — plus an apps/desktop Tauri app. The core crate has zero host dependencies and already targets wasm32, enabling the same audited engine to run in a future cloud worker. The roadmap lists JavaScript rendering for SPAs, crawl-to-crawl comparison and regression alerts, and an internal-link graph visualization.

Update: v0.4.0

The latest release is v0.4.0, published on 2026-06-22. The repository was created on 2026-06-18 and last pushed on 2026-06-22, indicating rapid early development. The project's GitHub topics include aeo, crawler, geo, marketing-tools, mcp, and seo, reflecting its dual focus on traditional SEO and AI-answer-engine optimization.