FrontierMemory

FrontierMemory is a benchmark from Letta for evaluating continual learning through memory. Agents read past trajectories and write a memory folder, and that memory is scored by how much it helps a fresh agent solve downstream tasks. The project is open source under Apache-2.0 and includes a command-line tool, a public leaderboard for models and harnesses, and a submission process.

What It Is

FrontierMemory is an evaluation framework for memory generation. The site says existing memory benchmarks mostly measure recall of past facts, while FrontierMemory measures memory quality by downstream task success. The README describes 15 memory-dependent tasks adapted from public coding and knowledge-work benchmarks, which test whether agents learn task contracts, user preferences, and environment rules from trajectories. The site names HarborIndex, DeepSWE, and SkillsBench among the task sources.

How a Run Works

A run has two stages. In memory generation, a memory agent reads a fixed corpus of 674 agent trajectories, each ending in a PASS or FAIL verifier result, and writes a memory folder. Generation is limited to six hours, and the agent has no web access beyond its model API. In the task stage, a task agent attempts the 15 target tasks with that memory, alongside a no-memory control. The score is the strict pass rate on each task's verifier.

The frontier-memory CLI, built on Harbor, covers the workflow:

• fetch-corpus downloads the gated corpus.
• generate-memory runs a memory harness.
• run-tasks evaluates task agents, with or without memory.
• score and bundle package results for submission.

Execution is supported on Docker or Modal. Python 3.12+ is required.

Leaderboard and Submissions

The leaderboard ranks models and harnesses by score and task cost, including Letta Dream, OpenAI Agents SDK, Prime Agent, Hermes Agent, and Claude Managed Agents. Official runs use Letta Dream for memory generation and Letta Code for task evaluation by default. Developers can submit a model by email, or a memory-generation harness through an adapter pull request. Submissions need three independent runs, which Letta verifies before posting an official score. Task-only submissions are not accepted.