# LayoutLens

> AI-powered visual UI testing library combining deterministic WCAG accessibility audits via axe-core with optional natural-language visual checks through vision LLMs.

LayoutLens is an open-source Python library for AI-assisted UI testing, published under the MIT License by the gojiplus organization. It combines deterministic browser-based checks — vendored axe-core WCAG 2.1 A/AA audits, geometry scorers, contrast analysis, and text-occlusion detection — with an optional vision-LLM tier powered by LiteLLM. The library is available on PyPI and targets Python 3.11+, with Playwright used for screenshot capture.

## What It Is

LayoutLens is a UI testing framework that operates across three tiers: a fully keyless deterministic tier (axe-core + geometry/contrast scorers), a hybrid tier where deterministic findings ground and can override LLM verdicts, and a pure LLM tier for natural-language visual questions. The deterministic tier requires no API key and is designed as a reliable CI gate; the LLM tier accepts any LiteLLM-compatible provider including self-hosted Ollama and vLLM endpoints. The project reports 81.1% measured accuracy on its bundled benchmark suite (60/74 labeled queries, gpt-4o-mini, measured 2026-07-21).

## Architecture: Three-Tier Testing Model

The library's design separates concerns across three distinct tiers:

- **Deterministic tier** — vendored axe-core 4.10.3 runs WCAG 2.1 A/AA checks against a real Playwright-rendered page; `LayoutScorer` detects contrast failures, element overlap, clipping, viewport protrusion, undersized targets (WCAG 2.5.8), focus obscuration (WCAG 2.4.11), and text occlusion. No API key or network call required; confidence is always 1.0.
- **Hybrid tier** (default for accessibility checks) — axe-core findings are injected into the LLM prompt as grounding context; any measured violation deterministically forces the verdict to "no" regardless of the model's answer.
- **LLM tier** — natural-language questions answered from a Playwright screenshot via any vision model; the `judge()` method sends prompts verbatim for use as an external evaluation harness.

## Integration Points

LayoutLens integrates across several developer workflows:

- **pytest plugin** — registers automatically on install; `assert_a11y` and `assert_layout` are keyless; `assert_ui` skips (never fails) without an API key or with `--layoutlens-no-llm`
- **CLI** — `layoutlens page.html --a11y axe` for keyless scans; `--output sarif` emits SARIF 2.1.0 for GitHub Code Scanning upload via `github/codeql-action/upload-sarif`
- **GitHub Actions** — a packaged `gojiplus/layoutlens-action@v1` bundles install, scan, PR annotations, a sticky results comment, and SARIF upload
- **MCP server** — `layoutlens[mcp]` exposes `audit_accessibility`, `scan_layout`, `check_ui`, and `compare_ui` as MCP tools for Claude Code, Cursor, and compatible coding agents
- **LiteLLM** — any provider supported by LiteLLM works, including OpenAI, Gemini, and self-hosted models via `api_base`

## Benchmark and Evaluation Design

The bundled benchmark covers 18 HTML fixtures and 74 labeled yes/no queries across four categories: responsive design (95.2% accuracy), layout alignment (79.2%), accessibility (76.2%), and UI components (62.5%). The evaluator scores answers deterministically — leading yes/no token vs. the answer key; ambiguous answers count as incorrect. The committed artifact `benchmarks/results/2026-07-21_gpt-4o-mini.json` is a real measured run. For a larger external benchmark (4,000+ machine-verified items), the README points to the companion UIJudgeBench project on GitHub and Hugging Face.

The README explicitly documents limitations: vision LLMs miss fine-grained UI differences (citing arXiv 2605.29615 showing frontier models at 47.2% overall on DiffSpot), axe-core passing is not full WCAG conformance, and the bundled benchmark is intentionally small and narrow.

## Update: v2.2.0

The latest release is v2.2.0, published 2026-08-18, with the repository last pushed 2026-08-25. A notable breaking change introduced in v1.7.0 requires every YAML test-suite case to declare `expected_results` (an `answer` field and/or a `contains` list); cases without it now raise `ValidationError` at load time. The batch judging API (`judge_batch`) supports provider-native async Batch APIs for OpenAI, Gemini, and LiteLLM file-based backends, with content-addressed resume manifests and per-manifest locking. Manifests created before v2.1.1 fail closed with explicit migration details.

## Features
- Deterministic axe-core WCAG 2.1 A/AA accessibility audits (keyless)
- Deterministic layout scorers: contrast, overlap, clipping, viewport protrusion, target size, focus obscuration, text occlusion
- Natural-language visual testing via vision LLMs (LiteLLM-compatible)
- Hybrid mode: deterministic findings ground and can override LLM verdicts
- Multi-viewport support: mobile, tablet, desktop
- pytest plugin with keyless assert_a11y and assert_layout assertions
- CLI with SARIF 2.1.0 output for GitHub Code Scanning
- MCP server exposing audit_accessibility, scan_layout, check_ui, compare_ui tools
- GitHub Actions integration via gojiplus/layoutlens-action
- Batch testing with concurrent async processing
- YAML/JSON test suite support
- LLM judge API with verbatim prompt passthrough
- Batch judging with provider-native async Batch APIs (OpenAI, Gemini, LiteLLM)
- 6 built-in domain experts: accessibility, conversion, mobile, ecommerce, healthcare, finance
- Self-hosted model support via api_base (Ollama, vLLM)
- Structured JSON output with TypedDict schemas
- Bundled benchmark suite (18 fixtures / 74 labeled queries)

## Integrations
OpenAI (gpt-4o, gpt-4o-mini), LiteLLM, Playwright, axe-core 4.10.3, Ollama, vLLM, Google Gemini, GitHub Actions, GitHub Code Scanning (SARIF), Claude Code (MCP), Cursor (MCP), pytest, PyPI

## Platforms
CLI, API, DEVELOPER_SDK

## Pricing
Open Source

## Version
v2.2.0

## Links
- Website: https://pypi.org/project/layoutlens/
- Documentation: https://gojiplus.github.io/layoutlens/
- Repository: https://github.com/gojiplus/layoutlens
- EveryDev.ai: https://www.everydev.ai/tools/layoutlens