SIE: Superlinked Inference Engine

SIE (Superlinked Inference Engine) is an open-source inference server built by Superlinked that runs the full range of models needed by AI agents — encoders, rerankers, entity extractors, OCR, and generation models — through a single OpenAI-compatible API on your own infrastructure. It is licensed under Apache 2.0 and available on GitHub with over 3,200 stars. The project targets teams who want to eliminate per-token API costs and keep prompts and documents inside their own cloud.

What It Is

SIE is a self-hosted Kubernetes inference cluster designed specifically for the multi-model workloads that power agents and search pipelines. Unlike LLM inference tools built for one large model spread across many GPUs, SIE solves the opposite problem: running many small models (encoders, rerankers, extractors) on shared GPUs with fast on-demand switching. It exposes four core primitives:

• Encode — converts text or images to vectors for semantic search and RAG
• Score — reranks query-document pairs for higher-precision retrieval
• Extract — pulls entities and structured data from unstructured text
• Generate — runs text generation on open LLMs you host yourself

The server handles batching, GPU sharing via LRU eviction, and model switching automatically. Over 100 models are supported out of the box, including bge-m3, SPLADE, ColBERTv2, Qwen3, GLiNER, and Granite Guardian.

Architecture: Pool-Then-Batch GPU Efficiency

SIE's core architectural differentiator is a cluster-wide queue that routes work before batching, rather than committing requests to individual workers first. According to the project's own benchmarks, this approach achieves 89% GPU efficiency versus 51% for solutions with worker-local queues. The stateless gateway publishes work to one pool queue; worker pods pull from it, form full batches, and share GPUs across many models. The engine wraps multiple backends — PyTorch, SGLang, Flash Attention, Candle (Rust), and Apple MLX — and selects the best backend per model automatically.

Worker pools can be configured for different workload profiles (real-time agent calls, nightly pipelines, eval suites) and scale to zero via KEDA autoscaling on Kubernetes.

Deployment Model

SIE is designed to run identically from a laptop to a production cluster. Deployment options include:

• Local: pip install "sie-server[local]" or Docker on CPU/GPU
• Kubernetes: Helm chart with KEDA autoscaling and Grafana dashboards
• Cloud Terraform modules: maintained separately for AWS EKS, GCP GKE, Azure AKS, and Alibaba Cloud ACK
• Air-gapped: installs from mirrored model snapshots

The project is SOC2 Type 2 certified. A managed hosted option and an agent plugin (for routing document work off frontier-model bills) are listed as upcoming on the website.

Integrations and Ecosystem

SIE publishes an OpenAI-compatible API (/v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses), enabling drop-in migration from OpenAI. Framework integrations include LangChain, LlamaIndex, Haystack, DSPy, and CrewAI. Vector store integrations cover Chroma, Qdrant, Weaviate, and LanceDB. An MCP edge package (sie_mcp) lets Claude and other MCP clients offload document parsing, extraction, and summarization to a self-hosted SIE cluster. Python and TypeScript SDKs are available via PyPI (sie-sdk) and npm (@superlinked/sie-sdk).

Update: v0.7.3

The latest release is v0.7.3, published on September 3, 2026. The repository was last pushed to on September 10, 2026, indicating active development. The project launched publicly with a blog post explaining the decision to open-source the inference engine, and the team has published a series of comparison posts positioning SIE against TEI, vLLM, Modal, FastEmbed, and OpenAI for different workload profiles. The Superlinked about page states the company raised $12M+ from Index Ventures, Theory Ventures, Samsung Next, and others, and the team includes ex-Google and Mastercard engineers.