# Safe-RLHF (Beaver)

> Open-source modular framework for SFT, RLHF and Safe RLHF training of language models, with datasets and evaluation tools.

Beaver (the `safe-rlhf` repository) is an open-source RLHF framework developed by the PKU-Alignment team at Peking University. It provides training code and human-labeled preference data for alignment research, with a focus on constrained value alignment through Safe RLHF. The code is released under the Apache License 2.0.

## What It Is

Safe-RLHF is a training pipeline for aligning large language models. It covers supervised fine-tuning (SFT), reward and cost model training, standard RLHF with PPO, and Safe RLHF with a Lagrangian PPO variant that maximizes reward subject to a cost constraint. The goal is a model that is both helpful (high reward) and harmless (low cost). The project's own comparison table lists DeepSpeed as the backend.

## Workflow and Outputs

Users clone the repository and set up a conda environment or a Docker container. Shell scripts then run each stage in order: SFT, reward model, cost model, PPO, and PPO-Lag. The outputs are fine-tuned checkpoints plus reward and cost models. The README documents tests with LLaMA-7B on 8 NVIDIA A800-80GB GPUs, and describes DeepSpeed ZeRO-Offload and multi-node options for setups with less GPU memory. Other supported base models include OPT, Baichuan and InternLM, and custom datasets can be added through a RawDataset abstraction.

## Datasets, Models and Evaluation

The project publishes the PKU-SafeRLHF dataset on Hugging Face. It is a human-labeled set of helpfulness and harmlessness preferences covering more than ten constraint dimensions. The README also lists released Beaver-7B models with matching reward and cost model checkpoints. For evaluation, the repository includes an interactive CLI demo, an arena comparing two models, reward/cost-model arena evaluation, BIG-bench tasks and GPT-4 evaluation. The authors say Beaver improves harmlessness over Alpaca in their GPT-4 evaluation.

## Features
- SFT, RLHF and Safe RLHF training pipelines
- Reward model and cost model training with pre-trained checkpoints
- PKU-SafeRLHF human-labeled preference dataset
- Custom dataset abstraction for SFT, preference and RL stages
- DeepSpeed backend with ZeRO-Offload and multi-node training
- Support for LLaMA, OPT, Baichuan and InternLM models
- Interactive CLI demo and arena
- Evaluation via BIG-bench, GPT-4 and reward/cost arena

## Integrations
DeepSpeed, Hugging Face, Weights & Biases, BIG-bench, OpenAI API, Docker

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Links
- Website: https://pku-beaver.github.io
- Documentation: https://github.com/PKU-Alignment/safe-rlhf#readme
- Repository: https://github.com/PKU-Alignment/safe-rlhf
- EveryDev.ai: https://www.everydev.ai/tools/safe-rlhf