# Colossal-AI

> An open-source distributed deep learning framework that maximizes runtime performance for large neural networks using advanced parallelism techniques.

Colossal-AI is an open-source distributed training framework designed to help researchers and engineers train large-scale neural networks with unmatched speed and efficiency. It provides a rich set of parallelism strategies—including tensor, pipeline, and data parallelism—that can be combined to maximize GPU utilization across clusters. The framework is developed by HPC-AI Technology and is actively maintained with a growing community of contributors and users.

- **Distributed Training** — *supports data, tensor, and pipeline parallelism out of the box, enabling efficient scaling across multiple GPUs and nodes.*
- **Hybrid Parallelism** — *combine multiple parallelism paradigms (e.g., train GPT with hybrid parallelism) to achieve optimal throughput for your specific model architecture.*
- **Gemini Heterogeneous Memory Manager** — *intelligently manages CPU and GPU memory to reduce out-of-memory errors and allow training of larger models on limited hardware.*
- **Command Line Interface (CLI)** — *a unified CLI tool to launch distributed jobs, run tensor parallel micro-benchmarks, and manage Colossal-AI projects.*
- **Flexible Configuration** — *define project configurations declaratively, specifying features, parallelism strategies, and global hyper-parameters in a single config file.*
- **Quick Start & Examples** — *get started quickly with installation guides, quick demos, and a rich library of usage examples covering common large model training scenarios.*
- **Active Community** — *engage with other users and contributors via GitHub Discussions, Slack, and the project forum; submit your own Colossal-AI projects to the showcase.*
- **Open Source** — *the full source code is publicly available on GitHub under an open-source license, making it freely usable and extensible for research and production.*

## Features
- Distributed training with data, tensor, and pipeline parallelism
- Hybrid parallelism for large model training
- Gemini heterogeneous memory manager
- Command Line Interface (CLI) for distributed job management
- Tensor parallel micro-benchmarking
- Flexible declarative configuration
- Support for large language model training (e.g., GPT)
- Usage examples and tutorials
- GitHub Discussions community forum
- Slack community

## Integrations
PyTorch, CUDA, NVIDIA GPUs

## Platforms
CLI, API, DEVELOPER_SDK

## Pricing
Open Source, Free tier available

## Links
- Website: https://colossalai.org
- Documentation: https://colossalai.org/docs/get_started/installation
- Repository: https://github.com/hpcaitech/ColossalAI
- EveryDev.ai: https://www.everydev.ai/tools/colossal-ai