xiaozhi-esp32-server

xiaozhi-esp32-server is an open-source backend service for the xiaozhi-esp32 smart hardware project, published by xinnan-tech under the MIT license. It implements the Xiaozhi communication protocol using Python, Java and Vue, so users with ESP32 devices can run their own server. The README credits development to Professor Siyuan Liu's team at South China University of Technology.

What It Is

The project is a self-hosted voice-interaction backend for ESP32 devices. It supports MQTT+UDP and WebSocket connections, MCP access points, voiceprint recognition and a knowledge base. It is intended for people who already own ESP32 hardware and want to run their own backend instead of using a hosted one.

Voice and Model Pipeline

The server chains streaming ASR, VAD, LLM, optional vision LLM and TTS. Documented components include FunASR and SherpaASR for local recognition, SileroVAD, and many TTS providers. LLMs connect through OpenAI-compatible interfaces, Ollama, Dify, FastGPT, Coze, Xinference and Home Assistant. Intent recognition runs via function calling or an LLM, memory options include local short-term memory, mem0ai and PowerMem, and RAGFlow supplies the knowledge base. 3D-Speaker handles multi-user voiceprint registration and speaker identification.

Deployment Options

There are two paths. The minimal install runs conversation and single-agent management from configuration files without a database. The full-module install adds multi-user and multi-agent management and a web console, with data stored in a database. Both can run via Docker or from source. The README lists 2 cores/4 GB for the minimal install with local FunASR (2 cores/2 GB with all-API components) and 4 cores/8 GB for the full install with local FunASR (2 cores/4 GB all-API). Test tools measure ASR, LLM, VLLM and TTS response speed.

Tools and Extensibility

Tool calling covers client IoT, client MCP, server MCP and MCP access-point protocols plus custom functions. A plugin system supports hot loading. MQTT lets the console push MCP commands to devices. The console UI supports Simplified Chinese, Traditional Chinese and English.

Current Status

The README warns that functionality is incomplete and has not passed network security evaluation, so it should not be used in production. The latest listed release is v0.9.7.