OpenCLIP

OpenCLIP is an open source implementation of OpenAI's CLIP (Contrastive Language-Image Pre-training), maintained under the mlfoundations organization. It provides pretrained image-text models and a training codebase, installable as the open_clip_torch Python package. The latest listed release is v3.3.0.

What It Is

OpenCLIP is a PyTorch library for loading, running, and training contrastive vision-language models. Users create a model, preprocessing transforms, and tokenizer, then encode images and text into embeddings and compare them for zero-shot classification. Pretrained weights come from datasets such as LAION-400M, LAION-2B and DataComp-1B, and it can also load other models such as SigLIP, DFN and PE-Core variants.

Training and Evaluation Workflow

The training entry point (open_clip_train.main) reads image-text data from CSV files or WebDataset tar shards and supports multiple data sources with upsampling factors. It scales from a single process to multi-node and SLURM runs, with options for gradient accumulation, patch dropout, model distillation, and checkpoint resume. Zero-shot evaluation can run on ImageNet or on audio datasets for CLAP models. The README also documents CoCa training and caption generation, int8 inference via bitsandbytes (beta), and pushing models to the Hugging Face Hub.

Main Branch Changes

The README notes that main uses a refactored training stack built on TrainingTask wrappers, dict-based batches, FSDP2 support, NaFlex variable-resolution pipelines, and several torch.compile strategies. New experimental model families include NaFlex CLIP, CLAP, GenLIP/GenLAP, MaMMUT and CoCa v2 configs. It lists breaking changes, including removal of Horovod and TorchScript support and a minimum of torch 2.6. Users who need the older training API are directed to the v3 branch or the 3.x PyPI releases.