LocalGPT

LocalGPT is an open-source, fully private document intelligence platform built by PromtEngineer, licensed under MIT and hosted on GitHub with over 22,000 stars. It lets users ask questions, summarize, and extract insights from their own documents using local AI models—no data ever leaves the device. The project was created in May 2023 and remains actively maintained, with the latest push recorded in July 2026.

What It Is

LocalGPT is a self-hosted RAG (Retrieval-Augmented Generation) system that runs entirely on your own hardware. Unlike cloud-based document Q&A tools, it routes all inference through locally running models via Ollama, keeping sensitive documents completely private. It goes beyond basic RAG with a hybrid search engine that blends semantic vector search, BM25 keyword matching, and Late Chunking for long-context precision. A smart router automatically decides whether to use RAG or direct LLM answering for each query, and an independent verification pass adds an extra accuracy layer.

Architecture and Pipeline

LocalGPT is built with a modular, four-service architecture managed by a single launcher script:

• Ollama Server (port 11434): handles local AI model serving
• RAG API Server (port 8001): document processing and retrieval
• Backend Server (port 8000): session management and REST API endpoints
• Frontend Server (port 3000): React/Next.js web interface

The retrieval pipeline supports two modes: a production-ready default pipeline with hybrid search, AI reranking (ColBERT), query decomposition, contextual enrichment, and answer verification; and a fast pipeline that strips overhead for speed-optimized use cases. The vector store is LanceDB, with BM25 handled separately. SQLite backs session and chat history storage.

Key Capabilities

• Hybrid search: combines dense vector search (weight 0.7) and BM25 sparse search (weight 0.3) with configurable reranking via ColBERT or BGE models
• Contextual enrichment: AI-generated context added to chunks at index time, inspired by Anthropic's Contextual Retrieval approach
• Query decomposition: complex queries are broken into sub-questions and answers composed back together
• Semantic caching: TTL-based cache with similarity matching for faster repeated queries
• Answer verification: a separate verification model pass checks generated answers for accuracy
• Multi-format document support: PDF, DOCX, TXT, Markdown (PDF fully supported; others in progress)
• RESTful API: complete programmatic access for building custom RAG applications
• Streaming responses: real-time chat output via streaming endpoints
• Hardware flexibility: supports CUDA, CPU, Intel Gaudi HPU, and Apple MPS backends

Deployment and Setup

LocalGPT offers three installation paths. Docker deployment is the simplest: clone the repo, install Ollama locally, pull models (qwen3:0.6b and qwen3:8b are recommended), and run the provided shell script. Direct development setup uses Python 3.8+ and Node.js 16+, with run_system.py orchestrating all four services. Manual component startup is also supported for fine-grained control. The README notes that installation is currently tested primarily on macOS, with Ubuntu/Debian and Windows instructions provided. System requirements call for 8 GB RAM minimum, with 16 GB recommended.

Update: LocalGPT v2 and Active Development

The repository's localgpt-v2 branch represents a significant architectural upgrade over the original version, introducing the hybrid search engine, smart routing, contextual enrichment, Late Chunking, ColBERT reranking, and the multi-service launcher. The README instructs users to clone the localgpt-v2 branch before it is merged to main. The project was last pushed in July 2026, indicating ongoing active development. The star history shows strong community traction since the original 2023 launch.