UQLM

UQLM is an open-source Python library from CVS Health for detecting hallucinations in large language model outputs using uncertainty quantification. Each scorer returns a confidence score between 0 and 1, where higher values indicate a lower likelihood of errors or hallucinations. It is installed from PyPI and works with LangChain chat models.

What It Is

UQLM is a developer library that wraps an LLM, generates responses to prompts, and scores how reliable each response is. Results are returned through a generate_and_score workflow and can be exported to a DataFrame. It also includes a mitigation option in which the response with the lowest uncertainty is selected.

Scorer Categories

The documentation groups scorers into five types:

• Black-box scorers measure consistency across multiple generations and work with any LLM. Examples include semantic entropy, non-contradiction probability, exact match, BERTScore and cosine similarity.
• White-box scorers use token probabilities for single-generation scoring and need access to logprobs. They include minimum token probability, sequence probability and P(True).
• LLM-as-a-judge scorers use one or more LLMs to evaluate a response, with categorical, continuous, Likert and panel-of-judges variants.
• Ensemble scorers combine scorers by weighted averaging. They can be used off the shelf or tuned on prompts with ground-truth answers.
• Long-text scorers score individual claims or sentences and can refine a response by removing low-confidence claims.

Code Generation and Long-Form Output

CodeGenUQ predicts whether LLM-generated code is functionally correct without executing it. It combines white-box scorers, code-adapted black-box scorers based on functional equivalence, and reflexive self-evaluation scorers. LongTextUQ decomposes a response into claims, scores each against sampled responses, and returns claim-level data along with a refined response.

Documentation and Learning Resources

The documentation site includes a quickstart, scorer definitions, an API reference, example notebooks, FAQs and release notes. The project is released under the Apache 2.0 license and is described in a JMLR software paper, with further TMLR and EMNLP papers on the methods.