Giskard Open Source

Giskard Open Source is a Python library from Giskard AI for testing and evaluating agentic systems. Version 3 is a rewrite built around modular packages, with a checks library for evals and a scan package for vulnerability and RAG quality scanning. It is released under the Apache 2.0 license and requires Python 3.12 or newer.

What It Is

Giskard wraps any target, whether an LLM, a black-box agent, or a multi-step pipeline, and tests it with scenarios and checks. Evals are designed for non-deterministic outputs, where the same input can produce different valid responses. The library sits alongside the commercial Giskard Hub, which the docs describe as an enterprise platform with a UI and SDK.

Giskard Checks

giskard-checks provides a scenario API in which a target is called through interactions and the traces are validated by checks. Built-in checks include string matching, comparisons, regex, semantic similarity, and LLM-as-judge checks such as Groundedness, Conformity, and LLMJudge. Scenarios can be grouped into suites and can cover multi-turn conversations. The README lists use cases such as catching regressions, validating RAG grounding, and enforcing safety rules.

Giskard Scan

giskard-scan is installed with the scan extra. vulnerability_scan generates adversarial test suites from a plain-language description of the agent, covering prompt injection, harmful content, stereotypes, and misinformation, and it supports custom scenario generators. quality_scan evaluates knowledge-base quality for RAG systems. The docs note that scan results are not a safety or compliance guarantee, and that LLM judges and generators need a provider extra and an API key.

Version Notes

Giskard v2 remains available but is no longer actively maintained. The legacy automatic scan for tabular and ML models exists only in v2. Telemetry is optional and aggregated, it sends no prompts or outputs, and it can be disabled with an environment variable.