Evidently

Evidently is an open-source Python library from Evidently AI for evaluating, testing, and monitoring ML and LLM systems, from experiments through production. It works with tabular and text data and ships 100+ built-in metrics, from data drift detection to LLM judges. The latest listed release is v0.7.23.

What It Is

Evidently is an ML and LLM observability framework distributed as a Python package. Users compute evaluations over datasets, summarize them in reports, and optionally track results over time in a monitoring UI. It supports both predictive and generative tasks, from classification to RAG, and lets users add custom metrics through a Python interface.

Reports and Test Suites

Reports compute and summarize data, ML and LLM quality evaluations using presets and built-in metrics, or custom ones. They can be viewed interactively in Python or exported as JSON, a Python dictionary, or HTML, or viewed in the monitoring UI. Adding pass/fail conditions (such as greater-than or less-than) turns a Report into a Test Suite for regression testing, CI/CD checks, or data validation. Test conditions can also be auto-generated from a reference dataset.

What You Can Evaluate

Documented evaluation areas include:

• Text descriptors such as length, sentiment, toxicity, and regex matches
• LLM outputs such as semantic similarity and retrieval relevance
• Data quality, including missing values and duplicates
• Data distribution drift, using 20+ statistical tests and distance metrics
• Classification and regression metrics
• Ranking and RAG metrics
• Recommendation metrics

Monitoring Dashboard

A Monitoring UI service visualizes metrics and test results over time. It can be self-hosted from the open-source version (for example with the evidently ui command and demo projects at localhost:8000) or used through Evidently Cloud, which the README says adds dataset and user management, alerting, and no-code evals.