LayoutLens

LayoutLens is an open-source Python library for AI-assisted UI testing, published under the MIT License by the gojiplus organization. It combines deterministic browser-based checks — vendored axe-core WCAG 2.1 A/AA audits, geometry scorers, contrast analysis, and text-occlusion detection — with an optional vision-LLM tier powered by LiteLLM. The library is available on PyPI and targets Python 3.11+, with Playwright used for screenshot capture.

What It Is

LayoutLens is a UI testing framework that operates across three tiers: a fully keyless deterministic tier (axe-core + geometry/contrast scorers), a hybrid tier where deterministic findings ground and can override LLM verdicts, and a pure LLM tier for natural-language visual questions. The deterministic tier requires no API key and is designed as a reliable CI gate; the LLM tier accepts any LiteLLM-compatible provider including self-hosted Ollama and vLLM endpoints. The project reports 81.1% measured accuracy on its bundled benchmark suite (60/74 labeled queries, gpt-4o-mini, measured 2026-07-21).

Architecture: Three-Tier Testing Model

The library's design separates concerns across three distinct tiers:

• Deterministic tier — vendored axe-core 4.10.3 runs WCAG 2.1 A/AA checks against a real Playwright-rendered page; LayoutScorer detects contrast failures, element overlap, clipping, viewport protrusion, undersized targets (WCAG 2.5.8), focus obscuration (WCAG 2.4.11), and text occlusion. No API key or network call required; confidence is always 1.0.
• Hybrid tier (default for accessibility checks) — axe-core findings are injected into the LLM prompt as grounding context; any measured violation deterministically forces the verdict to "no" regardless of the model's answer.
• LLM tier — natural-language questions answered from a Playwright screenshot via any vision model; the judge() method sends prompts verbatim for use as an external evaluation harness.

Integration Points

LayoutLens integrates across several developer workflows:

• pytest plugin — registers automatically on install; assert_a11y and assert_layout are keyless; assert_ui skips (never fails) without an API key or with --layoutlens-no-llm
• CLI — layoutlens page.html --a11y axe for keyless scans; --output sarif emits SARIF 2.1.0 for GitHub Code Scanning upload via github/codeql-action/upload-sarif
• GitHub Actions — a packaged gojiplus/layoutlens-action@v1 bundles install, scan, PR annotations, a sticky results comment, and SARIF upload
• MCP server — layoutlens[mcp] exposes audit_accessibility, scan_layout, check_ui, and compare_ui as MCP tools for Claude Code, Cursor, and compatible coding agents
• LiteLLM — any provider supported by LiteLLM works, including OpenAI, Gemini, and self-hosted models via api_base

Benchmark and Evaluation Design

The bundled benchmark covers 18 HTML fixtures and 74 labeled yes/no queries across four categories: responsive design (95.2% accuracy), layout alignment (79.2%), accessibility (76.2%), and UI components (62.5%). The evaluator scores answers deterministically — leading yes/no token vs. the answer key; ambiguous answers count as incorrect. The committed artifact benchmarks/results/2026-07-21_gpt-4o-mini.json is a real measured run. For a larger external benchmark (4,000+ machine-verified items), the README points to the companion UIJudgeBench project on GitHub and Hugging Face.

The README explicitly documents limitations: vision LLMs miss fine-grained UI differences (citing arXiv 2605.29615 showing frontier models at 47.2% overall on DiffSpot), axe-core passing is not full WCAG conformance, and the bundled benchmark is intentionally small and narrow.

Update: v2.2.0

The latest release is v2.2.0, published 2026-08-18, with the repository last pushed 2026-08-25. A notable breaking change introduced in v1.7.0 requires every YAML test-suite case to declare expected_results (an answer field and/or a contains list); cases without it now raise ValidationError at load time. The batch judging API (judge_batch) supports provider-native async Batch APIs for OpenAI, Gemini, and LiteLLM file-based backends, with content-addressed resume manifests and per-manifest locking. Manifests created before v2.1.1 fail closed with explicit migration details.