RagaAI Catalyst

RagaAI Catalyst is a Python SDK and platform from raga-ai-hub for managing and optimizing LLM projects and agentic AI applications. It covers projects, datasets, evaluations, traces, prompts, synthetic data, guardrails, and red-teaming. The repository is published under the Apache 2.0 license, and its latest listed release is v2.2.4.

What It Is

RagaAI Catalyst is an observability, monitoring, and evaluation framework for LLM and agent applications. After installing it with pip and authenticating with an access key and secret key generated from profile settings, developers use Python classes to organize work into projects and datasets, run metric evaluations, and record traces of their applications.

Tracing and Evaluation

The Tracer records runs of RAG apps and agents, either as a context manager or with explicit start and stop calls. Agentic tracing tracks LLM interactions and token usage, tool use, network activity and API calls, user interactions, and agent decision-making. It also includes cost tracking and performance monitoring utilities, plus auto-instrumentation. The Evaluation class applies metrics such as Faithfulness and Hallucination to a dataset with configurable models and thresholds, and it can append metrics for newly added rows.

Prompts, Synthetic Data, and Safety

PromptManager lists prompts, fetches specific versions, reads variables, and compiles prompts for use with OpenAI or LiteLLM. The synthetic data module generates Q&A pairs and examples from documents or CSV files. GuardrailsManager attaches guardrails such as response evaluators and regex checks to deployments, and GuardExecutor evaluates them at call time.

Red-Teaming

The red-teaming module scans applications for vulnerabilities, biases, and misuse. It supports multiple LLM providers, built-in and custom detectors, user-supplied or auto-generated test cases, and detailed reports. Results can optionally be uploaded to the Catalyst dashboard.