# OpenCLIP

> Open source PyTorch implementation of OpenAI's CLIP for training and running contrastive language-image models.

OpenCLIP is an open source implementation of OpenAI's CLIP (Contrastive Language-Image Pre-training), maintained under the mlfoundations organization. It provides pretrained image-text models and a training codebase, installable as the `open_clip_torch` Python package. The latest listed release is v3.3.0.

## What It Is

OpenCLIP is a PyTorch library for loading, running, and training contrastive vision-language models. Users create a model, preprocessing transforms, and tokenizer, then encode images and text into embeddings and compare them for zero-shot classification. Pretrained weights come from datasets such as LAION-400M, LAION-2B and DataComp-1B, and it can also load other models such as SigLIP, DFN and PE-Core variants.

## Training and Evaluation Workflow

The training entry point (`open_clip_train.main`) reads image-text data from CSV files or WebDataset tar shards and supports multiple data sources with upsampling factors. It scales from a single process to multi-node and SLURM runs, with options for gradient accumulation, patch dropout, model distillation, and checkpoint resume. Zero-shot evaluation can run on ImageNet or on audio datasets for CLAP models. The README also documents CoCa training and caption generation, int8 inference via bitsandbytes (beta), and pushing models to the Hugging Face Hub.

## Main Branch Changes

The README notes that `main` uses a refactored training stack built on TrainingTask wrappers, dict-based batches, FSDP2 support, NaFlex variable-resolution pipelines, and several torch.compile strategies. New experimental model families include NaFlex CLIP, CLAP, GenLIP/GenLAP, MaMMUT and CoCa v2 configs. It lists breaking changes, including removal of Horovod and TorchScript support and a minimum of torch 2.6. Users who need the older training API are directed to the `v3` branch or the 3.x PyPI releases.

## Features
- Pretrained CLIP-style image-text models loadable by name
- Zero-shot image classification
- Training on CSV and WebDataset data
- Multi-GPU, multi-node and SLURM distributed training
- FSDP2 and torch.compile strategies
- CoCa captioning and text generation
- CLAP audio-text training and zero-shot evaluation
- NaFlex variable-resolution image pipelines
- Model distillation and gradient accumulation
- Int8 inference via bitsandbytes (beta)
- Push models to Hugging Face Hub

## Integrations
PyTorch, timm, Hugging Face Hub, Hugging Face Transformers, WebDataset, img2dataset, CLIP_benchmark, bitsandbytes, TensorBoard, Weights & Biases

## Platforms
API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v3.3.0

## Links
- Website: https://github.com/mlfoundations/open_clip
- Repository: https://github.com/mlfoundations/open_clip
- EveryDev.ai: https://www.everydev.ai/tools/openclip