Knowhere

Knowhere, built by Ontos AI, ingests complex documents and turns them into structured, navigable context for AI agents. It is available as a managed cloud API and as an Apache 2.0 open-source backend that can be self-hosted. Outputs keep links to the original document, section, page, and region.

What It Is

Knowhere is a document parsing and retrieval system for agentic RAG, vector-based RAG, and other LLM workflows. It runs parsing, hierarchy reconstruction, multi-modal structuring, and graph construction in a single pipeline. The result is persistent memory that agents can navigate and cite, rather than flat text chunks.

Supported inputs include PDF, DOC/DOCX, XLS/XLSX, PPTX, JPG/PNG, Markdown, TXT, HTML, JSON, and CSV (per the website). EPUB, XML, MP4, MP3, and SKILL.md are listed as coming soon.

Dual-Track Parsing

Knowhere 2.0 uses two parsing tracks. The Text Track keeps precise extracted text and native structure where it is reliable. The Vision Track uses vision models to interpret a page or slide as a whole, so visually complex content stays recallable without perfect element-by-element extraction. The README says PDF and .pptx uploads through the V2 Jobs API use the Vision Track, and other formats use the Text Track. Both tracks produce the same chunk and metadata schema.

Agentic Retrieval and Citations

Knowhere exposes one corpus schema and tools for document outlines, structural filters, exact search, fuzzy recall, full reading, assets, and cross-document relationships. The agent decides how to explore. The same foundation serves built-in retrieval and external agents through MCP. Results resolve to documents, section paths, page numbers, and rendered visual evidence. Tables and images are summarized with VLM assistance and linked to their source sections.

Integration and Deployment

The cloud service is reachable through REST APIs, webhooks, Python and Node.js SDKs, and MCP, which the site says can bring context into Cursor, VS Code, Claude, or Codex. Users submit a URL or file, then receive structured JSON via webhook or polling. The open-source repository provides the API and worker, and separate repositories cover the dashboard and a Docker Compose self-hosted stack. Self-hosting needs Python 3.11+, uv, Docker, a database, Redis, S3-compatible storage, and at least one LLM provider key. A vision-capable model is needed for V2 PDF/PowerPoint parsing.

Performance Claims

The README reports internal evaluations stating +36% first-try accuracy and +11% recall over raw documents. It also reports 79% accuracy with feedback versus a ~53% ceiling on raw documents. These are vendor-published figures.