VibeVoice

VibeVoice is a family of open-source frontier voice AI models developed by Microsoft Research, covering both Text-to-Speech (TTS) and Automatic Speech Recognition (ASR). It uses continuous speech tokenizers operating at an ultra-low frame rate of 7.5 Hz and a next-token diffusion framework combining a Large Language Model with a diffusion head for high-fidelity audio generation. The project is released under the MIT License and is intended for research and development purposes.

Key models and features include:

• VibeVoice-ASR — A unified speech-to-text model that handles up to 60-minute long-form audio in a single pass, producing structured transcriptions with speaker identity (Who), timestamps (When), and content (What). Supports 50+ languages and customized hotwords.
• VibeVoice-TTS — A long-form multi-speaker TTS model capable of synthesizing up to 90 minutes of speech with up to 4 distinct speakers. Supports English, Chinese, and other languages with expressive, natural-sounding output.
• VibeVoice-Realtime-0.5B — A lightweight 0.5B parameter real-time streaming TTS model with ~300ms first-audible latency, supporting streaming text input and robust long-form generation (~10 minutes).
• Hugging Face Integration — All model weights are available on Hugging Face Hub; VibeVoice-ASR is natively supported via the Hugging Face Transformers library.
• vLLM Inference Support — VibeVoice-ASR supports vLLM for accelerated inference.
• Finetuning Support — Finetuning code for VibeVoice-ASR is publicly available in the repository.
• Google Colab Demos — Interactive Colab notebooks are provided for quick experimentation with streaming TTS and realtime models.
• Next-Token Diffusion Architecture — Core innovation using acoustic and semantic tokenizers at 7.5 Hz for efficient long-sequence processing while preserving audio fidelity.