Purple Llama

Purple Llama is an umbrella project from Meta that brings together safeguards and evaluation benchmarks for building responsibly with open generative AI models. It currently covers cybersecurity evaluations and input/output safeguards, hosted in the meta-llama GitHub organization.

What It Is

Purple Llama takes its name from purple teaming, which combines attack (red team) and defensive (blue team) postures. The project packages two kinds of components: system-level safeguards that filter what goes into and comes out of an LLM, and benchmarks that measure the cybersecurity risks of LLMs.

System-Level Safeguards

• Llama Guard 3 is a series of input and output moderation models fine-tuned from Meta-Llama 3.1 and 3.2. They detect content in the MLCommons standard hazards taxonomy. The README says they also support 7 new languages, a 128k context window and image reasoning, and are optimized to detect helpful cyberattack responses and prevent malicious code output from being executed in code-interpreter environments.
• Prompt Guard protects LLM-powered applications from prompt injection (untrusted third-party data that triggers unintended instructions) and jailbreaks (instructions that override a model's built-in safety features).
• Code Shield filters insecure code produced by LLMs at inference time. It addresses insecure code suggestions, code interpreter abuse and secure command execution, and includes an example notebook.

Evals and Benchmarks

The CyberSec Eval suites are based on industry guidance such as CWE and MITRE ATT&CK.

• CyberSec Eval v1 provides metrics for quantifying LLM cybersecurity risks and tools to evaluate insecure code suggestions and compliance with malicious requests.
• CyberSec Eval 2 adds tests for code interpreter abuse, offensive cybersecurity capabilities and susceptibility to prompt injection, with a Hugging Face leaderboard.
• CyberSec Eval 3 adds visual prompt injection, spear phishing capability and autonomous offensive cyber operations test suites.

Licensing and Getting Started

Evals and benchmarks, along with Code Shield, are MIT licensed. Llama Guard and Prompt Guard models use the corresponding Llama Community licenses. Meta recommends checking all LLM inputs and outputs against content guidelines suited to the application. Safeguards are being integrated as a safety layer in the Llama reference system, and getting-started resources are in the llama-recipes repository.