Hugging Face Datasets

Hugging Face Datasets is an open-source Python library from Hugging Face for accessing and sharing datasets used in machine learning. It loads datasets from the Hugging Face Hub or local files with a single load_dataset() call and provides processing and streaming methods to prepare data for training and evaluation. The latest release listed on GitHub is 5.1.0, and the project is licensed under Apache 2.0.

What It Is

Datasets is a data-loading and pre-processing library for audio, computer vision and natural language processing workloads. Its API centers on load_dataset(), which returns a dataset ready for use with NumPy, Pandas, PyTorch, TensorFlow, JAX or Polars. It also reads local CSV, JSON, JSONL, Parquet, HDF5, XML, text, image, audio, PDF and NIfTI files, and can build datasets from Python dictionaries, lists, Pandas frames or generators.

How the Data Workflow Works

The library is backed by Apache Arrow, which gives zero-copy, memory-mapped reads so large datasets are not limited by RAM. Processing happens through map(), with batched operation and num_proc multiprocessing, and results are cached so the same processing is not repeated. For datasets larger than disk, streaming=True iterates over data without downloading it. Two core classes, Dataset and IterableDataset, are wrapped in DatasetDict and IterableDatasetDict for multi-split data.

Modalities and Hub Integration

Built-in support covers text, audio, image, video, PDF and 3D medical (NIfTI) data, and the README also lists loading AI agent traces from the Hub. Datasets integrates with the Hugging Face Hub for discovering and sharing datasets, and can read from and write to Hub Storage Buckets. FAISS and Elasticsearch indexes enable similarity search. The README says streaming is up to 100x faster with the Xet backend.

Setup Path

Install with pip install datasets or conda from the huggingface channel. Optional extras add audio, vision, PDF/NIfTI and framework support (torch, tensorflow, jax). The documentation offers tutorials, how-to guides, conceptual guides and API reference, and users are asked to pin the revision of dataset repositories for reproducibility.