LatentSync
Open-source lip-sync framework using audio-conditioned latent diffusion models with SyncNet supervision.
At a Glance
Self-hosted LatentSync lip-sync code, inference, data processing and training code released under the Apache License 2.0.
Engagement
Available On
Alternatives
Listed Nov 2026
About LatentSync
LatentSync is an open-source lip-sync method from ByteDance that generates lip-synced video from audio using audio-conditioned latent diffusion, built on Stable Diffusion. The repository provides inference code, checkpoints, a data processing pipeline, and training code. Version 1.6, released 2025/06/11, is trained on 512x512 videos to reduce blurriness.
What It Is
LatentSync is an end-to-end lip-sync approach that models audio-visual correlations directly in latent space, without any intermediate motion representation. The README contrasts this with earlier diffusion-based lip-sync methods that use pixel-space diffusion or two-stage generation. Whisper converts the mel-spectrogram into audio embeddings, which are fed into a U-Net through cross-attention layers. The accompanying paper introduces StableSyncNet and a Temporal Representation Alignment (TREPA) mechanism to improve lip-sync accuracy and temporal consistency.
Running Inference
Users set up the environment and download checkpoints with a setup script, then generate lip-synced videos through either a Gradio app or a command-line script. Adjustable parameters include inference steps (20-50), where higher values improve visual quality but slow generation, and guidance scale (1.0-3.0), where higher values improve lip-sync accuracy but may cause distortion or jitter. Minimum VRAM for inference is documented as 8 GB for version 1.5 and 18 GB for version 1.6.
Training and Evaluation Tooling
The repository includes a data processing pipeline that removes broken videos, resamples video to 25 FPS and audio to 16000 Hz, detects scenes, splits clips, aligns faces, and filters by sync confidence and hyperIQA scores. U-Net training uses several config files with documented VRAM needs from 20 GB to 55 GB, and SyncNet can be trained on custom datasets. Evaluation scripts compute sync confidence scores and SyncNet accuracy.
Release Updates
Version 1.5 (2025/03/14) added a temporal layer for better temporal consistency, improved performance on Chinese videos, and lowered stage 2 training VRAM to 20 GB.
Community Discussions
Be the first to start a conversation about LatentSync
Share your experience with LatentSync, ask questions, or help others learn from your insights.
Pricing
Open Source (Apache-2.0)
Self-hosted LatentSync lip-sync code, inference, data processing and training code released under the Apache License 2.0.
- Apache License 2.0
- Inference code and checkpoints
- Data processing pipeline
- Training code
- Gradio app and CLI inference
Capabilities
Key Features
- Audio-conditioned latent diffusion lip-sync
- Whisper audio embeddings via cross-attention
- StableSyncNet supervision
- TREPA temporal consistency
- Gradio app and CLI inference
- Adjustable inference steps and guidance scale
- Data processing pipeline
- U-Net and SyncNet training scripts
- Sync confidence and SyncNet accuracy evaluation
- 512x512 resolution model (v1.6)
