# LoongForge

> An open-source high-performance training framework for LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs, with up to 5× speedup over open-source baselines.

LoongForge is an open-source training framework developed by the Baidu AI Cloud Baige team, released under the Apache 2.0 license. It delivers accelerated training for mainstream LLMs, VLMs, diffusion models, and embodied (VLA/WAM) models, with ready-to-run configs for 35+ model families built on Megatron-LM and torch-native backends. The project was publicly released on GitHub in April 2026 and is described as the open-source successor to AIAK-Training-LLM, Baidu's internal training acceleration suite.

## What It Is

LoongForge is a distributed model training framework targeting researchers and ML engineers who need to train large-scale foundation models efficiently. It covers four major model categories — Large Language Models (LLMs), Vision-Language Models (VLMs), diffusion models, and embodied AI models (VLA and world-action models) — under a single unified codebase. The framework adopts a multi-backend architecture: a patched Megatron-LM stack for LLMs, VLMs, and diffusion models, and a standalone torch-native DDP/FSDP subsystem for embodied models. It natively supports both NVIDIA GPUs and Baidu Kunlun XPUs via a minimally-intrusive plugin design.

## Architecture and Performance Optimizations

LoongForge's performance gains come from deep, per-model-family optimizations layered on top of its backends:

- **MoE EP Communication Optimization** — Overlapped All2All, activation offload, and compute for MoE models like DeepSeek-V3 and Qwen3-MoE.
- **Topology-Aware Optimal Transport (TAOT)** — Dynamic expert replica placement that tackles expert-parallel load imbalance, with the team reporting up to 74% lower overhead than industry solutions and a 1.43× speedup on a real MoE training case (published as an arXiv paper in August 2026).
- **Adaptive FP8 Training** — Per-operator FP8 decisions based on GEMM shape and efficiency.
- **Custom Fused Operators** — FusedDSA and Sparse MLA kernels; a TileLang version is open-sourced.
- **Heterogeneous Parallelism** — Independent TP/DP/recompute/freeze per model component (e.g., ViT vs. LLM backbone) to eliminate encoder-induced pipeline bubbles.
- **Delta-FP8 FSDP Communication** — Compresses BF16 FSDP2 AllGather deltas into blockwise FP8 on supported NVIDIA GPUs.
- **ChunkPipe** — Chunked long-sequence pipelining targeting million-length contexts.

Benchmark speedups reported by the project over open-source baselines include: DeepSeek-V3.2 Lite at 5.04×, DreamZero WAM at 4.38×, Pi0.5 VLA at 2.80×, GR00T N1.6 VLA at 2.31×, and Qwen3-VL-30B-A3B at 1.45×. The project notes these numbers were measured at a point in time and may evolve.

## Supported Models and Coverage

LoongForge ships ready-to-run configs and launch scripts for a broad model matrix:

- **LLMs**: DeepSeek-V2/V3/V3.2/V4, LLaMA2/3/3.1, Qwen through Qwen3-Next, MiniMax M2.1–M2.7, MIMO, GLM-5/5.2/5.3-flash, Kimi-K2.x/K3
- **VLMs**: Qwen2.5-VL, Qwen3-VL, Qwen3.5/3.6/3.8, Kimi-K2.5/K2.6/K3, MiniCPM-V-4.6, GLM-5.2+MoonViT, ERNIE4.5-VL, LLaVA-OneVision-1.5, InternVL2.5/3.5, and a CustomCombinedModel that composes any ViT + LLM via YAML
- **Diffusion**: Wan2.1, Wan2.2, Qwen-Image-Edit-2511
- **Embodied**: Pi0.5, GR00T-N1.6/N1.7, xVLA, Wall-OSS-0.5, FastWAM, LingBot-VA, Cosmos3, DreamZero

Training stages covered include pretrain, continued pre-training, SFT, and LoRA. Checkpoint conversion supports bidirectional Megatron ↔ HuggingFace format.

## Hardware and Deployment Model

LoongForge runs on NVIDIA GPUs (via the Megatron + TransformerEngine ecosystem) and Baidu Kunlun XPUs (via an XPU Plugin mechanism). The project states that production runs have reached 5,000+ XPUs on the predecessor AIAK-Training-LLM platform. Installation is Docker-first — a unified prebuilt image covers all model families — with source build also supported. The repository is structured with separate `examples/` and `examples_xpu/` directories for GPU and XPU launch scripts respectively.

## Update: v0.2.0 and Recent Activity

The latest tagged release is **v0.2.0**, published on August 20, 2026. Recent additions logged in the changelog include:

- GLM-5.3-flash and Kimi-K3 training support (September 2026)
- An optimized DreamZero Wan2.2-5B FSDP recipe with cache-aware data loading and Delta-FP8 AllGather (September 2026)
- Wall-OSS-0.5 VLA training with custom fused operators (August 2026)
- The TAOT paper release and 1.43× MoE speedup (August 2026)
- A unified evaluation module for the embodied stack covering Pi0.5, xVLA, and GR00T (August 2026)
- Unified prebuilt Docker images for all model families (July 2026)
- LoongForge-Embodied torch-native subsystem launch (July 2026)

The first official tagged release (v0.1.0) was in May 2026, shortly after the source code was made public in April 2026. The project has 27 contributors and 572 GitHub stars as of the data snapshot.

## Features
- Ready-to-run configs for 35+ model families
- Multi-backend architecture: Megatron-LM and torch-native
- MoE EP communication optimization with overlapped All2All
- Topology-Aware Optimal Transport (TAOT) for expert load balancing
- Adaptive FP8 training with per-operator precision decisions
- Custom fused operators (FusedDSA, Sparse MLA kernels)
- Heterogeneous parallelism per model component (TP/DP/recompute)
- Delta-FP8 FSDP communication compression
- ChunkPipe for long-sequence pipelining
- DP load balancing for sequence-packing imbalance
- Pretrain, SFT, and LoRA training stages
- Bidirectional Megatron ↔ HuggingFace checkpoint conversion
- Native NVIDIA GPU and Kunlun XPU support
- Unified Docker image for all model families
- Embodied model training with DDP/ZeRO-1/FSDP/HSDP
- Unified evaluation module for embodied models (LIBERO, CALVIN, SimplerEnv, RoboTwin)
- CustomCombinedModel: compose any ViT + LLM via YAML
- Decoupled encoder-decoder training to eliminate pipeline bubbles
- Context Parallel (CP) for long-sequence LLM training
- INT4/NVFP4 PTQ support

## Integrations
Megatron-LM, HuggingFace Transformers, TransformerEngine, PyTorch, Docker, NVIDIA CUDA, Baidu Kunlun XPU, LLaMA-Factory, LeRobot, TileLang, Hydra (YAML config), DeepSeek models, Qwen models, LLaMA models, InternVL, LLaVA-OneVision, GR00T, OpenPI

## Platforms
CLI, API, LINUX

## Pricing
Open Source

## Version
v0.2.0

## Links
- Website: https://baidu-baige.github.io/LoongForge/
- Documentation: https://loongforge.readthedocs.io/en/latest/index.html
- Repository: https://github.com/baidu-baige/LoongForge
- EveryDev.ai: https://www.everydev.ai/tools/loongforge