llmware

llmware is an open-source Python framework from AI Bloks for building knowledge-based LLM applications that run locally, on edge devices or self-hosted. It combines a model catalog of 300+ models with an integrated RAG pipeline covering parsing, embedding, retrieval and prompting. It is distributed under the Apache-2.0 license and installs with pip.

What It Is

llmware is a developer toolkit for retrieval augmented generation and multi-step agent workflows. Its README says it is optimized for AI PCs, laptops, edge and self-hosted deployment on Windows, Mac and Linux. It supports GGUF, OpenVINO, ONNXRuntime, ONNXRuntime-QNN, WindowsLocalFoundry and PyTorch behind a common interface. Cloud models from OpenAI, Anthropic and Google are also supported.

How the RAG Pipeline Works

A Library is the knowledge-base container. Calling add_files routes mixed file types (pdf, pptx, docx, xlsx, txt, csv, md, json, wav, png, jpg, html) to parsers, which chunk the text and index it. Embeddings can be installed on a library with a chosen embedding model and vector database, and several can coexist. The Query class supports text, semantic, hybrid, metadata and custom-filter retrieval. Prompt with Sources packages retrieved passages or documents into model-ready context and offers post-inference evidence checks and prompt history.

Models and Agents

The catalog includes the llmware-finetuned SLIM, BLING, DRAGON and Industry-BERT series. SLIM models are fine-tuned for function calling, and the LLMfx agent class loads them as tools such as sentiment, extract, topics and boolean. It aggregates the results into a report and activity summary. Many models ship in quantized GGUF versions for CPU use.

Data Stores and Setup

Text collection databases are SQLite, MongoDB and Postgres. Supported vector databases are Milvus, PGVector, Neo4j, Redis, LanceDB, ChromaDB, Qdrant, FAISS, Pinecone and Mongo Atlas. A no-install option uses SQLite with ChromaDB. Docker Compose scripts are provided for the server-based options. OCR parsing needs optional Tesseract and Poppler packages.