Agent Memory Leaderboard

Agent Memory Leaderboard (AML) is an open evaluation platform launched on July 29, 2026 by researchers from more than twenty universities and research organizations. It provides a shared protocol, versioned evaluation process, and public leaderboard for measuring how well memory systems store, retrieve, and support the use of long-term information. The platform is free to participate in and is now operating as a continuous public evaluation program.

What It Is

AML is a benchmark leaderboard specifically designed for agent memory systems — the components that allow AI agents to persist, retrieve, and reason over long-term information. Unlike general LLM benchmarks, AML focuses exclusively on the memory layer: participants expose only Add and Search API endpoints, while the platform controls answer generation, scoring, aggregation, and run orchestration. This separation ensures that score differences primarily reflect the memory system itself rather than downstream model choices. The platform supports three evaluation tracks: Textual Memory, Multimodal Memory, and Coding Agent Memory, each with its own result table and metric breakdown.

How the Evaluation Protocol Works

The evaluation flow is designed to be controlled and reproducible:

• Participants deploy Add/Search APIs — publicly reachable endpoints that write memory chunks and return ranked retrieval results.
• The platform calls Add per sample and source session, then calls Search per question and receives ranked candidates.
• A fixed platform workflow generates answers using a locked answer model, scores them with fixed evaluators, and aggregates results.
• Results are private first, then eligible for public leaderboard publication after a review gate.

The textual benchmark suite covers capabilities including explicit fact recall, compositional inference, temporal and event reasoning, memory governance, personalization, context learning and execution, and safety and privacy. The coding track evaluates whether an agent can retrieve, filter, and reuse relevant engineering experience from earlier work in the same repository.

Benchmark Coverage and Datasets

The textual suite includes more than 10 benchmark datasets, over 1,500 histories and tasks, and nearly 5,000 evaluation questions, drawing from sources such as LoCoMo-Refined, ScriptMem, LongMemEval, CLBench, PersonaMem-v2, and BEAM. The coding track uses CAMBench Coding — 150 software-engineering tasks run under relevant and noisy memory conditions for 300 scored attempts across 12 repositories. The multimodal track covers ATM-Bench and Mem-Gallery, evaluating text-and-image memory retrieval. Two participant divisions exist: Open-source Methods (requiring public code and attribution) and Commercial Products (requiring a stable API; internal implementation may remain private).

Update: Cycle 2 and Current Status

The first leaderboard cycle opened July 29, 2026, with the first public leaderboard released August 12, 2026. The GitHub README notes that AML appeared near the top of the Hugging Face Trending Spaces page around August 20, 2026. The first cycle is now closed. Cycle 2 is scheduled to open September 20, 2026, with a materials deadline of October 31, 2026, evaluation closing November 4, 2026, and official results expected in mid-November 2026. The second cycle includes a prize pool of RMB 150,000 across Textual, Coding, and Multimodal tracks, available only to eligible Open-source Methods teams. The GitHub repository (AML-memory/agent-memory-leaderboard) had 1,180 stars and 68 forks as of mid-September 2026.

Co-organizer Network

The platform lists co-organizers from a broad set of institutions, including Tianjin University, Beijing University of Posts and Telecommunications, Jilin University, Nankai University, Xiamen University, UNSW Sydney, Nanjing University, Shanghai AI Laboratory, Southeast University, University of Oxford, Nanyang Technological University, Zhejiang University, Peking University, Renmin University of China, Fudan University, USTC, Tsinghua University, Shanghai Jiao Tong University, and others. Datawhale, an open-source AI community, is also listed as a co-organizer. These affiliations are as presented on the official website.

Tradeoffs and Participation Requirements

Participants must self-host their Add/Search APIs — AML does not deploy repository-only or Docker-only submissions. Each participant covers their own API, storage, bandwidth, and compute costs. Formal full evaluations are limited to two runs per AML Key per track per cycle, with the second run unlocking 30 days after the first. Smoke tests are capped at 30 per track per edition. The platform enforces strict sample isolation (no cross-user_id retrieval), synchronous Add semantics (HTTP 200 only after persistence), and a Top K of 100 for formal evaluations. Results that fail contract validation — such as missing required fields or exceeding Top K — are not silently truncated but immediately fail the evaluation stage.