SGLang-Omni

SGLang-Omni is an open-source serving runtime from the SGLang-Omni Team, built on top of SGLang, for omni, speech, and text-to-speech models. It splits generation into heterogeneous pipeline stages and exposes OpenAI-compatible endpoints for chat, speech, and transcription. The latest listed release is v0.1.8, available on PyPI.

What It Is

SGLang-Omni is a multi-stage serving runtime for models whose generation decomposes into stages with different compute profiles, such as preprocessing, encoders, autoregressive engines, talkers, decoders, and vocoders. It owns the pipeline topology, stage lifecycle, inter-stage transport, and model-family integration layer, while composing with SGLang for autoregressive scheduling and model execution where applicable. New models plug in by declaring a pipeline topology.

How the Pipeline Works

Each stage runs behind a scheduler matched to its workload, from SGLang-backed autoregressive scheduling to lightweight preprocessing and streaming vocoder loops. A control plane coordinates requests, while a relay data plane moves tensor payloads across shared-memory, NCCL, NIXL, and Mooncake backends. The docs also cover process topology, replicas, GPU sharing, and same-GPU data parallelism with CUDA MPS.

API Surface and Model Coverage

The server provides OpenAI-compatible /v1/audio/speech, /v1/audio/transcriptions, /v1/audio/translations, and /v1/chat/completions endpoints with streaming, batch speech, and uploaded voices. Documented models include TTS (Higgs Audio, Fish Audio S2-Pro and Drama 3, Voxtral, Qwen3-TTS, MOSS-TTS, ZONOS2 and others), music (MiniMax Music 3), ASR (Qwen3-ASR, Fun-ASR, Whisper, MOSS-Transcribe-Diarize and others), omni models (Qwen3-Omni, MiniCPM-o, Ming-Omni), and LLaDA2.0-Uni. A Rust-based router provides a multi-worker front door with health, readiness, and capability discovery.

Hardware and Setup

NVIDIA CUDA is the default backend with full model coverage. Apple Silicon (Qwen3-ASR via MLX or Torch MPS) and Intel XPU are marked experimental, and install guides exist for Ascend NPU, Intel CPU, and MUSA. Installation is documented via pip/uv from PyPI or an install script on macOS.