garak

garak (Generative AI Red-teaming & Assessment Kit) is an open-source LLM vulnerability scanner hosted under the NVIDIA GitHub organization. It checks whether a language model or dialog system can be made to fail in unwanted ways, probing for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks and other weaknesses. The README compares its role to tools like nmap or Metasploit, but for LLMs.

What It Is

garak is a command-line tool installed from PyPI or from source. You point it at a target model, and it runs probes against it, then applies detectors to judge whether the outputs show an undesirable behavior. It combines static, dynamic and adaptive probes to explore ways of making a model fail.

How a Scan Works

You choose a generator with --target_type and optionally --target_name, then run all probes by default or narrow them with --spec (for example a probe family such as promptinject or a single probe). Each probe declares a primary detector and extended detectors. For each probe, garak prints a progress bar and then a per-detector result row; any attempt that yields undesirable behavior is marked FAIL with a failure rate. Runs write a garak.log file, a structured JSONL report per run, and a hit log of successful attacks.

Targets and Probes

Supported targets include Hugging Face models (local pipeline, Inference API and private endpoints), Replicate, OpenAI, Cohere, Groq, AWS Bedrock, NVIDIA NIM, LiteLLM, GGUF/ggml models such as llama.cpp, and any REST endpoint configurable through a YAML file. Probe families include DAN-style jailbreaks, encoding-based prompt injection, GCG adversarial suffixes, glitch tokens, training-data replay, malware generation, package hallucination, XSS and data exfiltration, toxicity, and Language Model Risk Cards.

Extending It

The code is organized into probes, detectors, evaluators, generators and harnesses, each with base classes that plugins inherit from. Developers can write their own plugins and test them with blank generators and always-pass detectors. The project is released under the Apache 2.0 license, and a preprint paper describes the framework.