LMMs-Eval

LMMs-Eval is an evaluation toolkit from the non-profit research organization LMMs-Lab. It runs benchmarks for large multimodal models across text, image, video, and audio tasks through a single pipeline. The latest release listed on GitHub is v0.7.3.

What It Is

LMMs-Eval is a Python benchmarking framework forked from EleutherAI's lm-evaluation-harness and adapted for multimodal inputs. Users pick a model and one or more tasks, then run evaluations from the command line, a web UI, or an HTTP server. The README states three principles: reproducible, efficient, and trustworthy results, including confidence intervals, clustered standard errors, and paired comparisons.

Running Evaluations

The README documents a quickstart that installs the package with uv and evaluates a model such as Qwen2.5-VL on a benchmark like MME. Backends include vLLM, SGLang, Hugging Face model wrappers, and any OpenAI-compatible endpoint. The README says it covers 100+ tasks and 30+ model families, including MMMU, MME, MMBench, MathVista, and VideoMME. Environment variables such as OPENAI_API_KEY and HF_TOKEN are needed for some tasks.

Extending It

Chat models are the recommended interface, using structured messages with roles and interleaved text, image, video, and audio content. Legacy simple models use separate visual and text functions. New benchmarks are added through a YAML task configuration plus a doc_to_messages function.

Web UI and Eval Server

An optional web UI offers model and task selection, a live command preview, streaming output, and a log viewer. A separate HTTP evaluation server accepts jobs asynchronously via sync and async Python clients, so evaluations can run on dedicated GPU nodes while training continues. The README warns that the server is for trusted environments only.

Recent Releases

The README lists v0.7 (Feb 2026) as adding 25+ tasks, agentic task evaluation, a faster video I/O path with TorchCodec, safety baselines, and efficiency metrics. v0.6 introduced the standalone eval server and statistical results such as confidence intervals and paired t-tests.