esp32-pie-asr

esp32-pie-asr is an open-source project by Boumedine Billal that runs live English speech recognition on the ESP32-P4 microcontroller. The board's microphone feeds a streaming model and the recognized words appear on the board's screen as you speak, with no cloud. The firmware can be flashed from a browser page using Chrome or Edge.

What It Is

The project is embedded firmware plus an inference engine for the ESP32-P4X. It uses Kroko (by Banafo), a 65.7M-parameter streaming Zipformer2 transducer, with weights reduced from 71 MB to 26.65 MB using 2- and 4-bit quantization. Inference runs on the chip's two RISC-V cores using PIE SIMD instructions, with no neural accelerator. The firmware opens on live transcription and has a second page of six voice-control demos.

Recognition and Voice Control

The README reports 4.34% WER on LibriSpeech test-clean and 10.26% on test-other, at 0.70x real time, with words final about 2 seconds after they are spoken. A text file on the microSD card lists phrases (hotwords) to listen for, which can be toggled on screen without retraining. The six voice-control demos (robot vacuum, smart home, car cabin, factory line, kitchen, presenter) map spoken orders to simulated device actions using word rules rather than a language model.

Setup Path

The model file is copied to a FAT32 microSD card, since it does not fit the board's 16 MB flash. The browser installer then flashes the firmware over the USB Serial/JTAG port. It requires an ESP32-P4X-Function-EV-Board (chip revision v3). Alternatively, the firmware can be built with ESP-IDF v6.1. The code is Apache-2.0 and the weights are CC-BY-SA 4.0.