Liger Kernel

Liger Kernel is an open-source collection of Triton kernels for LLM training, developed by LinkedIn. It provides Hugging Face-compatible implementations of layers such as RMSNorm, RoPE, SwiGLU, CrossEntropy and FusedLinearCrossEntropy, plus post-training loss kernels. The project states it can raise multi-GPU training throughput by about 20% and reduce memory usage by about 60% under its benchmark conditions.

What It Is

Liger Kernel is a Python library that replaces standard model layers and loss functions with fused, memory-efficient Triton kernels. It is used by ML engineers who train or fine-tune LLMs and want longer context lengths, larger batch sizes or larger vocabularies on the same hardware. The authors state the computation is exact rather than approximate, with forward and backward passes covered by unit tests and convergence tests.

How You Apply It

There are three paths. AutoLigerKernelForCausalLM automatically patches supported Hugging Face models. Model-specific functions such as apply_liger_kernel_to_llama let you choose which kernels to enable. Individual modules, such as LigerFusedLinearCrossEntropyLoss, can also be composed into your own model. Patching APIs cover many model families, including Llama, Mistral, Gemma, Qwen, Phi3, DeepSeek and GPT-OSS.

Post-Training and Distillation Kernels

Chunked fused-linear loss kernels cover alignment methods (DPO, ORPO, CPO, SimPO, KTO) and distillation losses (JSD, KL divergence, TVD). The project says these can cut memory usage by up to 80%.

Hardware and Backends

The kernels run on CUDA and ROCm, with Ascend NPU dependencies also documented. Optional cuTile and CuTe DSL backends for CUDA are enabled via the LIGER_KERNEL_IMPL environment variable. The kernels work with Flash Attention, PyTorch FSDP, DeepSpeed and DDP. Trainer integrations listed include Axolotl, LLaMA-Factory, TRL SFTTrainer, Hugging Face Trainer, SWIFT and oumi.