# UQLM

> Python library for detecting LLM hallucinations using uncertainty quantification scorers that return confidence scores between 0 and 1.

UQLM is an open-source Python library from CVS Health for detecting hallucinations in large language model outputs using uncertainty quantification. Each scorer returns a confidence score between 0 and 1, where higher values indicate a lower likelihood of errors or hallucinations. It is installed from PyPI and works with LangChain chat models.

## What It Is

UQLM is a developer library that wraps an LLM, generates responses to prompts, and scores how reliable each response is. Results are returned through a `generate_and_score` workflow and can be exported to a DataFrame. It also includes a mitigation option in which the response with the lowest uncertainty is selected.

## Scorer Categories

The documentation groups scorers into five types:

- **Black-box scorers** measure consistency across multiple generations and work with any LLM. Examples include semantic entropy, non-contradiction probability, exact match, BERTScore and cosine similarity.
- **White-box scorers** use token probabilities for single-generation scoring and need access to logprobs. They include minimum token probability, sequence probability and P(True).
- **LLM-as-a-judge scorers** use one or more LLMs to evaluate a response, with categorical, continuous, Likert and panel-of-judges variants.
- **Ensemble scorers** combine scorers by weighted averaging. They can be used off the shelf or tuned on prompts with ground-truth answers.
- **Long-text scorers** score individual claims or sentences and can refine a response by removing low-confidence claims.

## Code Generation and Long-Form Output

`CodeGenUQ` predicts whether LLM-generated code is functionally correct without executing it. It combines white-box scorers, code-adapted black-box scorers based on functional equivalence, and reflexive self-evaluation scorers. `LongTextUQ` decomposes a response into claims, scores each against sampled responses, and returns claim-level data along with a refined response.

## Documentation and Learning Resources

The documentation site includes a quickstart, scorer definitions, an API reference, example notebooks, FAQs and release notes. The project is released under the Apache 2.0 license and is described in a JMLR software paper, with further TMLR and EMNLP papers on the methods.

## Features
- Black-box consistency-based scorers
- White-box token-probability scorers
- LLM-as-a-judge and LLM panel scorers
- Ensemble scorers with optional supervised tuning
- Long-text claim-level scoring with response refinement
- CodeGenUQ for code correctness prediction without execution
- Confidence scores between 0 and 1
- Works with any LangChain chat model
- Example notebooks

## Integrations
LangChain, OpenAI, Azure OpenAI, Google Vertex AI, Ollama, PyPI

## Platforms
LINUX, API, DEVELOPER_SDK

## Pricing
Open Source

## Version
v0.6.7

## Links
- Website: https://cvs-health.github.io/uqlm/latest/index.html
- Documentation: https://cvs-health.github.io/uqlm/latest/getstarted.html
- Repository: https://github.com/cvs-health/uqlm
- EveryDev.ai: https://www.everydev.ai/tools/uqlm