# fastCRW

> Web data API and open-source Rust engine that searches, scrapes, crawls, maps, and extracts web pages into clean markdown or structured JSON for AI agents.

fastCRW is a web data API and open-source engine from fastCRW (operated by Nanofy OÜ) that turns URLs into clean markdown or structured JSON for AI agents and pipelines. It covers search, scrape, crawl, map, extract, monitor, and PDF parsing, and can be used through a managed cloud API or run locally as a single Rust binary.

## What It Is

fastCRW is a scraping and web-data engine aimed at AI agents, RAG pipelines, and data workflows. The core engine is written in Rust and licensed AGPL-3.0, while the Python and TypeScript SDKs are MIT-licensed. It exposes a Firecrawl-compatible API on the overlap surface (/scrape, /crawl, /map, /search), and the docs note minor divergences in response field names and error envelopes.

## How the Workflow Operates

Each operation has a distinct output:

- Scrape turns one URL into markdown, HTML, links, screenshots, or schema-based JSON.
- Crawl follows a bounded site crawl and collects its pages.
- Map discovers URLs on a domain without scraping every page.
- Search queries the web and can optionally scrape selected results.
- Extract produces structured fields from one or many URLs using a prompt or JSON schema.
- Monitor watches a page for changes and delivers an AI verdict and a webhook.
- Parse converts PDFs to markdown, CPU-only and without an LLM.

The cloud service handles JavaScript rendering, anti-detection, proxies, and caching, and supports page actions such as click, scroll, write, and wait. Responses include source URLs and metadata.

## Agent and Developer Access

Users can call it from a CLI, a Python SDK (pip install crw), a Node.js SDK (npm install crw-sdk), REST, or an MCP server. A one-command installer (npx crw-mcp install) registers the skill and MCP server with detected tools such as Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf. Listed integrations include LangChain, CrewAI, Dify, n8n, Make, and Zapier.

## Managed vs Self-Hosted

The managed API provides proxies, browser rendering, and scaling with no infrastructure to run. The local or self-hosted option lets users choose renderers, search, auth, proxies, and capacity, with a documented Docker setup. Capabilities and response shapes can differ by deployment. The vendor publishes benchmark claims, including a 90.0% score on AA-Omniscience and the highest truth-recall on Firecrawl's 1,000-URL dataset, with a public harness for reproducing them. Crawl and map follow robots.txt by default.

## Features
- Web search with optional page content scraping
- Scrape URLs to markdown, HTML, links, screenshots, or JSON
- Bounded site crawling
- URL mapping without full page loads
- Structured extraction via prompt or JSON schema
- Page change monitoring with AI verdict and webhook
- PDF to markdown parsing (CPU-only, no LLM)
- JavaScript rendering and anti-detection
- Page actions: click, scroll, write, wait, press
- Source attribution in every response
- Official MCP server
- Python and Node.js SDKs and CLI
- Firecrawl-compatible API
- Self-hosting via single Rust binary or Docker

## Integrations
Claude Code, Cursor, Codex, Gemini CLI, OpenCode, Windsurf, LangChain, CrewAI, Dify, Agno, n8n, Make, Zapier, Cline, Continue.dev, Roo Code, Model Context Protocol

## Platforms
MACOS, LINUX, WEB, API, VSC_EXTENSION, DEVELOPER_SDK, CLI

## Pricing
Open Source, Free tier available

## Version
v0.37.2

## Links
- Website: https://fastcrw.com
- Documentation: https://docs.fastcrw.com
- Repository: https://github.com/fastcrw/crw
- EveryDev.ai: https://www.everydev.ai/tools/fastcrw