PageSift

PageSift is a self-hosted web and document extraction engine published by YOLOW92 on GitHub. It fetches a URL, rendering JavaScript only when that adds content, or opens a local file, and returns Markdown or structured JSON. The current release is 1.2.0, distributed as a single Rust binary with a CLI, HTTP API, MCP server and language packages.

What It Is

PageSift is a web scraper, crawler and document parser aimed at feeding LLMs, RAG indexes and agents. Inputs include URLs, HTML, PDFs (including scanned pages via OCR), Word, Excel and PowerPoint files, OpenDocument, e-books, email, archives, databases, source code and WARC archives. Outputs include Markdown, HTML, links, images, metadata and schema-based JSON, with warnings when a result is incomplete.

Interfaces and Compatibility

The project exposes a CLI, a Firecrawl v1/v2-compatible HTTP API, Jina Reader-style endpoints, and an MCP server over stdio or HTTP. Packages are listed for Python, Node.js/TypeScript and Rust, plus a C ABI with bindings for Go, Java, Kotlin, C#, Ruby, PHP and others, along with LangChain and LlamaIndex. It runs on Linux, macOS arm64 and Windows x86_64, and is also published as a Docker image.

Workflow Capabilities

• Structured extraction by CSS selectors that are relocated when a site changes, or by a schema written by your own model.
• Browser actions such as clicks, typing and scrolling, with persistent sessions.
• An agent endpoint that answers questions from the web using your own model and checks each field against the source evidence.
• Scheduled change monitoring with webhooks, sitemap and query-focused crawling, and WARC output.
• Multi-server gateway, worker and standby roles, with robots.txt, rate-limit and private-network guards.

Licensing and Update

PageSift is released under AGPL-3.0, with a separate commercial license offered for proprietary linking or private deployment. Version 1.2.0 adds optional, off-by-default access-barrier recovery with scoped sessions, validated proxy tunnels and a bounded wait for non-interactive challenges. Interactive CAPTCHAs are reported as needing a human, and no solver is included. The README states that standalone image-file OCR is not covered and that the tool is self-hosted rather than a managed cloud service, and it publishes its own benchmark comparisons against other tools.