Label Studio

Label Studio is an open source data labeling tool from HumanSignal, released under the Apache 2.0 license. It lets teams label audio, text, images, video, HTML, and time-series data through a configurable UI and export the results to various model formats. The project's site also positions it for AI evaluation and human-in-the-loop workflows. The latest listed version is 1.23.2.

What It Is

Label Studio is a multi-type annotation platform used to prepare raw data or improve existing training data for ML models. Users create projects, import data from files or cloud storage, and label it through interfaces built from a configuration language or from included templates. Multi-user support ties each annotation to the account that created it, and one instance can hold several projects.

Labeling Workflow and Data Handling

Data can be imported from files (JSON, CSV, TSV, RAR, ZIP) or synced from Amazon S3 and Google Cloud Storage. Labeling interfaces are customizable, and the documentation lists templates and customizable tags for common use cases. Annotations can be exported to formats for common ML libraries, including through the label-studio-converter.

ML Integration and Evaluation

The Machine Learning SDK connects a user-run ML backend server to a project. This supports pre-labeling with model predictions, online learning as annotations arrive, and active learning that focuses labeling on the most difficult examples. A REST API, Python SDK, and webhooks let it sit inside a data pipeline. The homepage also describes LLM and agent evaluation use cases such as reviewing agentic traces, collecting RLHF preferences, building custom benchmarks and rubrics, and grading RAG answers.

Setup Path

Label Studio installs with pip (Python 3.10 or later), Docker, Docker Compose (with Nginx and PostgreSQL), Poetry, or Anaconda. One-click deploys are offered for Heroku, Azure, and Google Cloud. The homepage says some interfaces, such as DICOM and LiDAR annotation, are part of Label Studio Enterprise, a separate commercial edition.