# Hugging Face Datasets

> Open-source Python library for loading, processing, streaming and sharing AI datasets for audio, vision and text tasks.

Hugging Face Datasets is an open-source Python library from Hugging Face for accessing and sharing datasets used in machine learning. It loads datasets from the Hugging Face Hub or local files with a single `load_dataset()` call and provides processing and streaming methods to prepare data for training and evaluation. The latest release listed on GitHub is 5.1.0, and the project is licensed under Apache 2.0.

## What It Is

Datasets is a data-loading and pre-processing library for audio, computer vision and natural language processing workloads. Its API centers on `load_dataset()`, which returns a dataset ready for use with NumPy, Pandas, PyTorch, TensorFlow, JAX or Polars. It also reads local CSV, JSON, JSONL, Parquet, HDF5, XML, text, image, audio, PDF and NIfTI files, and can build datasets from Python dictionaries, lists, Pandas frames or generators.

## How the Data Workflow Works

The library is backed by Apache Arrow, which gives zero-copy, memory-mapped reads so large datasets are not limited by RAM. Processing happens through `map()`, with batched operation and `num_proc` multiprocessing, and results are cached so the same processing is not repeated. For datasets larger than disk, `streaming=True` iterates over data without downloading it. Two core classes, `Dataset` and `IterableDataset`, are wrapped in `DatasetDict` and `IterableDatasetDict` for multi-split data.

## Modalities and Hub Integration

Built-in support covers text, audio, image, video, PDF and 3D medical (NIfTI) data, and the README also lists loading AI agent traces from the Hub. Datasets integrates with the Hugging Face Hub for discovering and sharing datasets, and can read from and write to Hub Storage Buckets. FAISS and Elasticsearch indexes enable similarity search. The README says streaming is up to 100x faster with the Xet backend.

## Setup Path

Install with `pip install datasets` or conda from the huggingface channel. Optional extras add audio, vision, PDF/NIfTI and framework support (torch, tensorflow, jax). The documentation offers tutorials, how-to guides, conceptual guides and API reference, and users are asked to pin the `revision` of dataset repositories for reproducibility.

## Features
- One-line dataset loading with load_dataset()
- Streaming mode without full downloads
- Apache Arrow zero-copy memory-mapped storage
- Smart caching of processed results
- Multi-processing with map(num_proc=N)
- Multi-modal support: text, audio, image, video, PDF, NIfTI
- Multiple formats: CSV, JSON, JSONL, Parquet, Arrow, XML, WebDataset
- FAISS and Elasticsearch search indexes
- Hugging Face Storage Buckets read/write
- AI agent trace loading
- Conversion to NumPy, Pandas, Polars, PyTorch, TensorFlow, JAX, Spark
- Share datasets to the Hugging Face Hub

## Integrations
Hugging Face Hub, PyTorch, TensorFlow, JAX, NumPy, Pandas, Polars, PyArrow, Spark, FAISS, Elasticsearch, Hugging Face Transformers, Hugging Face Storage Buckets

## Platforms
WEB, API, VSC_EXTENSION, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
5.1.0

## Links
- Website: https://huggingface.co/docs/datasets
- Documentation: https://huggingface.co/docs/datasets
- Repository: https://github.com/huggingface/datasets
- EveryDev.ai: https://www.everydev.ai/tools/hugging-face-datasets