ML Drift

ML Drift is an open-source GPU inference engine from the google-ai-edge organization, written in C++ and licensed under Apache 2.0. It runs machine learning workloads, including large generative models, on mobile phones, web browsers, and servers. The README describes it as the successor to the GPU delegate in TensorFlow Lite.

What It Is

ML Drift is a runtime for executing ML models on device GPUs. It targets developers who want low latency and low power use, plus privacy and offline operation, for apps on Android, iOS, the web, and GPU-equipped edge devices. Models are represented as a backend-agnostic GpuModel compute graph, and individual steps are GpuOperation units containing shader code. The README states that it achieves an order-of-magnitude performance improvement over existing open-source GPU inference engines.

How It Works

Kernels are written once in the Unified Compute Language (UCL), an abstraction over GLSL, MSL, and WGSL, and shaders are generated at runtime for the specific model, inputs, and GPU. Supported backends are OpenCL (primary for Android), Metal, WebGPU via Dawn, and OpenGL ES 3.1+. Runtime optimizations include operator fusion, layout adjustments, weight rearrangement, workgroup tuning, and memory reuse with algorithms such as GREEDY_BY_SIZE. Tensor virtualization decouples logical tensor views from physical GPU storage.

Large Generative Models

For LLM workloads, ML Drift provides stage-aware execution separating prefill from decode, efficient KV cache management, and FP16/INT8/INT4 quantization. GpuModel graphs can also be hand-tailored for custom models.

Getting Started

The repository includes hello-world guides for the OpenCL and WebGPU APIs and a samples directory. Public API headers live in api/, with backend implementations in cl/, gl/, metal/, and webgpu/.