Safe-RLHF (Beaver)

Beaver (the safe-rlhf repository) is an open-source RLHF framework developed by the PKU-Alignment team at Peking University. It provides training code and human-labeled preference data for alignment research, with a focus on constrained value alignment through Safe RLHF. The code is released under the Apache License 2.0.

What It Is

Safe-RLHF is a training pipeline for aligning large language models. It covers supervised fine-tuning (SFT), reward and cost model training, standard RLHF with PPO, and Safe RLHF with a Lagrangian PPO variant that maximizes reward subject to a cost constraint. The goal is a model that is both helpful (high reward) and harmless (low cost). The project's own comparison table lists DeepSpeed as the backend.

Workflow and Outputs

Users clone the repository and set up a conda environment or a Docker container. Shell scripts then run each stage in order: SFT, reward model, cost model, PPO, and PPO-Lag. The outputs are fine-tuned checkpoints plus reward and cost models. The README documents tests with LLaMA-7B on 8 NVIDIA A800-80GB GPUs, and describes DeepSpeed ZeRO-Offload and multi-node options for setups with less GPU memory. Other supported base models include OPT, Baichuan and InternLM, and custom datasets can be added through a RawDataset abstraction.

Datasets, Models and Evaluation

The project publishes the PKU-SafeRLHF dataset on Hugging Face. It is a human-labeled set of helpfulness and harmlessness preferences covering more than ten constraint dimensions. The README also lists released Beaver-7B models with matching reward and cost model checkpoints. For evaluation, the repository includes an interactive CLI demo, an arena comparing two models, reward/cost-model arena evaluation, BIG-bench tasks and GPT-4 evaluation. The authors say Beaver improves harmlessness over Alpaca in their GPT-4 evaluation.