hayamimi

hayamimi (早耳, Japanese for "quick ear") is an open-source, CPU-only real-time speech-to-text tool built by oboroge0. It delivers live subtitles with partial text appearing while you're still speaking, and finalized lines landing roughly 100ms after you stop — all under 2GB of RAM, with no GPU or cloud API dependency. The project reached v0.1.2 in August 2026 and has accumulated over 250 GitHub stars since its initial release.

What It Is

hayamimi is a multilingual automatic speech recognition (ASR) pipeline that routes each utterance to a specialist model based on detected language, rather than relying on a single general-purpose model like Whisper. It runs every model as a quantized INT8 ONNX model via sherpa-onnx, eliminating PyTorch and CUDA dependencies entirely. The result is a system that the project's own benchmarks show achieving 5.8% CER on real broadcast Japanese audio — less than half the 13.8% CER of whisper-large-v3-turbo on the same clips — while running at 10–50x realtime on a 6-core desktop CPU.

Language Routing Architecture

The core design decision is a 5-route language catalog, each backed by a best-in-class specialist model:

• Japanese: ReazonSpeech (Zipformer)
• Chinese (Mandarin): Paraformer-zh
• Korean / Cantonese: SenseVoice Small
• English + 24 EU languages: NVIDIA Parakeet TDT v3
• ~1,600 other languages: Meta Omnilingual ASR fallback

Language identification runs via a whisper-tiny spoken-LID model on the first ~4 seconds of each segment, with character-set arbitration as a secondary signal. An LRU model cache keeps resident models under a configurable memory cap (default: under 2GB total), evicting least-recently-used non-Japanese models as sessions wander across languages.

Key Features

• Partial subtitles: draft text updates every ~0.5s while speech is in progress
• Two-pass refinement: after 2 seconds of silence, recent utterances are batch re-decoded for a higher-accuracy "clean" transcript (improving Japanese CER from 15.5% to 12.0% in project benchmarks)
• Speaker labels: --speakers tags each utterance S1/S2/… using CAM++ speaker embeddings (turn-taking, not full diarization)
• Live translation: --translate en,zh,ko translates Japanese lines in real time (English via FuguMT, Chinese/Korean via M2M-100)
• OBS overlay + browser dashboard: --serve starts a local HTTP server with a browser-source overlay for stream captions and a live dashboard showing partials, finals, language badges, speaker chips, per-line latency, and the refined transcript
• Network audio input: --input ws accepts mic audio over WebSocket from a phone or ESP32 board, feeding it through the same pipeline
• Hotwords / user dictionary: --hotwords biases decoding toward proper nouns (note: currently has no effect on the Japanese tier due to a token encoding incompatibility); --replace does post-hoc find/replace and works everywhere

Measured Performance

Project-published benchmarks on real speech (single clips, no preroll/two-pass) show:

RTF well under 0.2 across every route means each route runs 9–16x faster than realtime on CPU alone. Mean final latency across a 5-language soak test with all features enabled is reported as ~236ms mean / 552ms max.

Update: v0.1.2 — User-Reported Fixes & Network Audio Input

The latest release, v0.1.2 (published August 26, 2026), is titled "user-reported fixes & network audio input." It follows the initial v0.1.0 release and adds the --input ws WebSocket ingest endpoint, enabling phones and ESP32/stackchan boards to stream mic audio over a LAN. The project is actively maintained with a detailed iteration log (docs/BENCHMARKS.md) covering 30+ measured changes tracking latency, memory, and accuracy tradeoffs.

Known Limitations

The project maintains an honest limitations list: code-switching mid-sentence is not supported (the router picks one language per utterance); --hotwords has no effect on the Japanese tier; --speakers does turn-taking labeling only, not true diarization; and translation quality has a real ceiling given the small model sizes used (FuguMT, M2M-100). The end-to-end mic pipeline has been tested primarily on Windows 11; macOS/Linux are expected to work but are not yet CI-tested end to end.