esp32-thinks

esp32-thinks is an open-source firmware recipe by raullenchai that turns a Seeed SenseCAP Watcher (ESP32-S3) into a personal voice assistant. Say "Hi, ESP" or press the knob, and the device answers out loud, streaming speech sentence by sentence while the model is still writing. It is built on Meta's muse-gadget-sdk and is released under the MIT license, with v0.2.12 as the latest listed release.

What It Is

The project is a patch and tooling set applied to a pinned commit of muse-gadget-sdk. Speech-to-text, chat and text-to-speech are three independent endpoints, each pointed at any OpenAI-compatible server, so one provider can serve all three or each task can use a different one. Presets are provided for openai, quicksilver, openmuse and custom setups. Replies follow the user's language, with English, Chinese and Korean tested.

How a Turn Works

A turn starts from the on-chip WakeNet10 wake word or the knob. The recording is trimmed to speech and uploaded to the STT endpoint, then the chat endpoint streams a response over SSE. The reply is split at sentence boundaries and each sentence is synthesized and appended to the audio already playing. The device keeps its TLS connection prepared, supports barge-in by pressing the knob during speech, discards silent recordings, and reboots after detected peripheral stalls.

Optional openmuse Agent

The repository includes an openmuse agent that sits between the device and an OpenAI-compatible LLM. It adds location, clock, weather and persistent memory, optional web search, owns the system prompt, and proxies STT so STT and chat share one TLS session. It can run via Docker or Node and can be exposed over HTTPS with Tailscale Funnel.

Setup Path

Setup requires ESP-IDF v6.0.x on Linux or macOS. Users clone the SDK and this repo, back up the factory NVS, run apply.sh with a preset to generate provider_config.h (keys, Wi-Fi, wake word), then run build-and-flash.sh. API keys are compiled into the firmware, which the README describes as a hobby recipe rather than a multi-tenant product. A serial console offers typed turns, status, task dumps and screenshots for bench testing.