# hayamimi

> Real-time multilingual speech-to-text on CPU only, with live subtitles, browser dashboard, speaker labels, and on-the-fly translation — no GPU or cloud required.

hayamimi (早耳, Japanese for "quick ear") is an open-source, CPU-only real-time speech-to-text tool built by oboroge0. It delivers live subtitles with partial text appearing while you're still speaking, and finalized lines landing roughly 100ms after you stop — all under 2GB of RAM, with no GPU or cloud API dependency. The project reached v0.1.2 in August 2026 and has accumulated over 250 GitHub stars since its initial release.

## What It Is

hayamimi is a multilingual automatic speech recognition (ASR) pipeline that routes each utterance to a specialist model based on detected language, rather than relying on a single general-purpose model like Whisper. It runs every model as a quantized INT8 ONNX model via [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx), eliminating PyTorch and CUDA dependencies entirely. The result is a system that the project's own benchmarks show achieving 5.8% CER on real broadcast Japanese audio — less than half the 13.8% CER of `whisper-large-v3-turbo` on the same clips — while running at 10–50x realtime on a 6-core desktop CPU.

## Language Routing Architecture

The core design decision is a 5-route language catalog, each backed by a best-in-class specialist model:

- **Japanese**: ReazonSpeech (Zipformer)
- **Chinese (Mandarin)**: Paraformer-zh
- **Korean / Cantonese**: SenseVoice Small
- **English + 24 EU languages**: NVIDIA Parakeet TDT v3
- **~1,600 other languages**: Meta Omnilingual ASR fallback

Language identification runs via a whisper-tiny spoken-LID model on the first ~4 seconds of each segment, with character-set arbitration as a secondary signal. An LRU model cache keeps resident models under a configurable memory cap (default: under 2GB total), evicting least-recently-used non-Japanese models as sessions wander across languages.

## Key Features

- **Partial subtitles**: draft text updates every ~0.5s while speech is in progress
- **Two-pass refinement**: after 2 seconds of silence, recent utterances are batch re-decoded for a higher-accuracy "clean" transcript (improving Japanese CER from 15.5% to 12.0% in project benchmarks)
- **Speaker labels**: `--speakers` tags each utterance S1/S2/… using CAM++ speaker embeddings (turn-taking, not full diarization)
- **Live translation**: `--translate en,zh,ko` translates Japanese lines in real time (English via FuguMT, Chinese/Korean via M2M-100)
- **OBS overlay + browser dashboard**: `--serve` starts a local HTTP server with a browser-source overlay for stream captions and a live dashboard showing partials, finals, language badges, speaker chips, per-line latency, and the refined transcript
- **Network audio input**: `--input ws` accepts mic audio over WebSocket from a phone or ESP32 board, feeding it through the same pipeline
- **Hotwords / user dictionary**: `--hotwords` biases decoding toward proper nouns (note: currently has no effect on the Japanese tier due to a token encoding incompatibility); `--replace` does post-hoc find/replace and works everywhere

## Measured Performance

Project-published benchmarks on real speech (single clips, no preroll/two-pass) show:

| Language | Route | Mean error | Mean RTF |
|---|---|---|---|
| Japanese | ReazonSpeech | 7.5% CER | 0.071 |
| English | Parakeet v3 | 2.3% WER | 0.109 |
| Chinese | Paraformer-zh | 5.3% CER | 0.102 |
| Korean | SenseVoice | 8.1% CER | 0.062 |
| Cantonese | SenseVoice | 6.1% CER | 0.061 |

RTF well under 0.2 across every route means each route runs 9–16x faster than realtime on CPU alone. Mean final latency across a 5-language soak test with all features enabled is reported as ~236ms mean / 552ms max.

## Update: v0.1.2 — User-Reported Fixes & Network Audio Input

The latest release, v0.1.2 (published August 26, 2026), is titled "user-reported fixes & network audio input." It follows the initial v0.1.0 release and adds the `--input ws` WebSocket ingest endpoint, enabling phones and ESP32/stackchan boards to stream mic audio over a LAN. The project is actively maintained with a detailed iteration log (`docs/BENCHMARKS.md`) covering 30+ measured changes tracking latency, memory, and accuracy tradeoffs.

## Known Limitations

The project maintains an honest limitations list: code-switching mid-sentence is not supported (the router picks one language per utterance); `--hotwords` has no effect on the Japanese tier; `--speakers` does turn-taking labeling only, not true diarization; and translation quality has a real ceiling given the small model sizes used (FuguMT, M2M-100). The end-to-end mic pipeline has been tested primarily on Windows 11; macOS/Linux are expected to work but are not yet CI-tested end to end.

## Features
- Real-time multilingual speech-to-text on CPU only
- 5-route language catalog (ja/zh/ko/yue/en+24 EU languages + ~1600-language fallback)
- Partial subtitles updating every ~0.5s while speaking
- Fast finals (~100ms after end of speech for Japanese)
- Two-pass refinement for higher-accuracy clean transcript
- Speaker labeling with CAM++ embeddings (--speakers)
- Live translation to English, Chinese, Korean (--translate)
- Browser dashboard with live subtitles, language badges, speaker chips, per-line latency
- OBS browser-source overlay for stream captions
- Network audio input via WebSocket (--input ws) for phone/ESP32
- Hotwords/user dictionary support (--hotwords, --replace)
- LRU model eviction for memory-bounded operation (<2GB RAM)
- No GPU, no PyTorch, no cloud API required
- Quantized INT8 ONNX models via sherpa-onnx
- Language identification via whisper-tiny spoken-LID
- Silero VAD for voice activity detection
- Transcript file output (--transcript)

## Integrations
OBS Studio (browser source overlay), sherpa-onnx (ONNX Runtime inference), ReazonSpeech (Japanese ASR), NVIDIA Parakeet TDT v3 (English + EU languages ASR), Paraformer-zh (Chinese ASR), SenseVoice Small (Korean/Cantonese ASR), Meta Omnilingual ASR (1600+ language fallback), FuguMT (Japanese-to-English translation), M2M-100 (Japanese-to-Chinese/Korean translation), Silero VAD (voice activity detection), CAM++ (speaker embeddings), Kiwi/kiwipiepy (Korean morphological tokenizer), ESP32/stackchan (WebSocket audio input), ffmpeg

## Platforms
WINDOWS, MACOS, LINUX, WEB, API, CLI

## Pricing
Open Source

## Version
v0.1.2

## Links
- Website: https://github.com/oboroge0/hayamimi
- Documentation: https://github.com/oboroge0/hayamimi
- Repository: https://github.com/oboroge0/hayamimi
- EveryDev.ai: https://www.everydev.ai/tools/hayamimi