LoongForge

LoongForge is an open-source training framework developed by the Baidu AI Cloud Baige team, released under the Apache 2.0 license. It delivers accelerated training for mainstream LLMs, VLMs, diffusion models, and embodied (VLA/WAM) models, with ready-to-run configs for 35+ model families built on Megatron-LM and torch-native backends. The project was publicly released on GitHub in April 2026 and is described as the open-source successor to AIAK-Training-LLM, Baidu's internal training acceleration suite.

What It Is

LoongForge is a distributed model training framework targeting researchers and ML engineers who need to train large-scale foundation models efficiently. It covers four major model categories — Large Language Models (LLMs), Vision-Language Models (VLMs), diffusion models, and embodied AI models (VLA and world-action models) — under a single unified codebase. The framework adopts a multi-backend architecture: a patched Megatron-LM stack for LLMs, VLMs, and diffusion models, and a standalone torch-native DDP/FSDP subsystem for embodied models. It natively supports both NVIDIA GPUs and Baidu Kunlun XPUs via a minimally-intrusive plugin design.

Architecture and Performance Optimizations

LoongForge's performance gains come from deep, per-model-family optimizations layered on top of its backends:

• MoE EP Communication Optimization — Overlapped All2All, activation offload, and compute for MoE models like DeepSeek-V3 and Qwen3-MoE.
• Topology-Aware Optimal Transport (TAOT) — Dynamic expert replica placement that tackles expert-parallel load imbalance, with the team reporting up to 74% lower overhead than industry solutions and a 1.43× speedup on a real MoE training case (published as an arXiv paper in August 2026).
• Adaptive FP8 Training — Per-operator FP8 decisions based on GEMM shape and efficiency.
• Custom Fused Operators — FusedDSA and Sparse MLA kernels; a TileLang version is open-sourced.
• Heterogeneous Parallelism — Independent TP/DP/recompute/freeze per model component (e.g., ViT vs. LLM backbone) to eliminate encoder-induced pipeline bubbles.
• Delta-FP8 FSDP Communication — Compresses BF16 FSDP2 AllGather deltas into blockwise FP8 on supported NVIDIA GPUs.
• ChunkPipe — Chunked long-sequence pipelining targeting million-length contexts.

Benchmark speedups reported by the project over open-source baselines include: DeepSeek-V3.2 Lite at 5.04×, DreamZero WAM at 4.38×, Pi0.5 VLA at 2.80×, GR00T N1.6 VLA at 2.31×, and Qwen3-VL-30B-A3B at 1.45×. The project notes these numbers were measured at a point in time and may evolve.

Supported Models and Coverage

LoongForge ships ready-to-run configs and launch scripts for a broad model matrix:

• LLMs: DeepSeek-V2/V3/V3.2/V4, LLaMA2/3/3.1, Qwen through Qwen3-Next, MiniMax M2.1–M2.7, MIMO, GLM-5/5.2/5.3-flash, Kimi-K2.x/K3
• VLMs: Qwen2.5-VL, Qwen3-VL, Qwen3.5/3.6/3.8, Kimi-K2.5/K2.6/K3, MiniCPM-V-4.6, GLM-5.2+MoonViT, ERNIE4.5-VL, LLaVA-OneVision-1.5, InternVL2.5/3.5, and a CustomCombinedModel that composes any ViT + LLM via YAML
• Diffusion: Wan2.1, Wan2.2, Qwen-Image-Edit-2511
• Embodied: Pi0.5, GR00T-N1.6/N1.7, xVLA, Wall-OSS-0.5, FastWAM, LingBot-VA, Cosmos3, DreamZero

Training stages covered include pretrain, continued pre-training, SFT, and LoRA. Checkpoint conversion supports bidirectional Megatron ↔ HuggingFace format.

Hardware and Deployment Model

LoongForge runs on NVIDIA GPUs (via the Megatron + TransformerEngine ecosystem) and Baidu Kunlun XPUs (via an XPU Plugin mechanism). The project states that production runs have reached 5,000+ XPUs on the predecessor AIAK-Training-LLM platform. Installation is Docker-first — a unified prebuilt image covers all model families — with source build also supported. The repository is structured with separate examples/ and examples_xpu/ directories for GPU and XPU launch scripts respectively.

Update: v0.2.0 and Recent Activity

The latest tagged release is v0.2.0, published on August 20, 2026. Recent additions logged in the changelog include:

• GLM-5.3-flash and Kimi-K3 training support (September 2026)
• An optimized DreamZero Wan2.2-5B FSDP recipe with cache-aware data loading and Delta-FP8 AllGather (September 2026)
• Wall-OSS-0.5 VLA training with custom fused operators (August 2026)
• The TAOT paper release and 1.43× MoE speedup (August 2026)
• A unified evaluation module for the embodied stack covering Pi0.5, xVLA, and GR00T (August 2026)
• Unified prebuilt Docker images for all model families (July 2026)
• LoongForge-Embodied torch-native subsystem launch (July 2026)

The first official tagged release (v0.1.0) was in May 2026, shortly after the source code was made public in April 2026. The project has 27 contributors and 572 GitHub stars as of the data snapshot.