# LatentSync

> Open-source lip-sync framework using audio-conditioned latent diffusion models with SyncNet supervision.

LatentSync is an open-source lip-sync method from ByteDance that generates lip-synced video from audio using audio-conditioned latent diffusion, built on Stable Diffusion. The repository provides inference code, checkpoints, a data processing pipeline, and training code. Version 1.6, released 2025/06/11, is trained on 512x512 videos to reduce blurriness.

## What It Is

LatentSync is an end-to-end lip-sync approach that models audio-visual correlations directly in latent space, without any intermediate motion representation. The README contrasts this with earlier diffusion-based lip-sync methods that use pixel-space diffusion or two-stage generation. Whisper converts the mel-spectrogram into audio embeddings, which are fed into a U-Net through cross-attention layers. The accompanying paper introduces StableSyncNet and a Temporal Representation Alignment (TREPA) mechanism to improve lip-sync accuracy and temporal consistency.

## Running Inference

Users set up the environment and download checkpoints with a setup script, then generate lip-synced videos through either a Gradio app or a command-line script. Adjustable parameters include inference steps (20-50), where higher values improve visual quality but slow generation, and guidance scale (1.0-3.0), where higher values improve lip-sync accuracy but may cause distortion or jitter. Minimum VRAM for inference is documented as 8 GB for version 1.5 and 18 GB for version 1.6.

## Training and Evaluation Tooling

The repository includes a data processing pipeline that removes broken videos, resamples video to 25 FPS and audio to 16000 Hz, detects scenes, splits clips, aligns faces, and filters by sync confidence and hyperIQA scores. U-Net training uses several config files with documented VRAM needs from 20 GB to 55 GB, and SyncNet can be trained on custom datasets. Evaluation scripts compute sync confidence scores and SyncNet accuracy.

## Release Updates

Version 1.5 (2025/03/14) added a temporal layer for better temporal consistency, improved performance on Chinese videos, and lowered stage 2 training VRAM to 20 GB.

## Features
- Audio-conditioned latent diffusion lip-sync
- Whisper audio embeddings via cross-attention
- StableSyncNet supervision
- TREPA temporal consistency
- Gradio app and CLI inference
- Adjustable inference steps and guidance scale
- Data processing pipeline
- U-Net and SyncNet training scripts
- Sync confidence and SyncNet accuracy evaluation
- 512x512 resolution model (v1.6)

## Integrations
Hugging Face, Replicate, Whisper, Stable Diffusion, Gradio

## Platforms
LINUX, API, CLI

## Pricing
Open Source

## Version
1.6

## Links
- Website: https://arxiv.org/abs/2412.09262
- Documentation: https://github.com/bytedance/LatentSync
- Repository: https://github.com/bytedance/LatentSync
- EveryDev.ai: https://www.everydev.ai/tools/latentsync
