SUPER Benchmark
A benchmark and agent code for evaluating LLM agents on setting up and executing ML/NLP tasks from research repositories.
At a Glance
About SUPER Benchmark
SUPER is a benchmark from allenai for evaluating LLM agents on setting up and executing tasks from research repositories on GitHub. The repository provides the benchmark resources along with agent code that can be run locally, in Docker, or on modal.com, and it is released under the Apache License 2.0.
What It Is
SUPER (Evaluating Agents on Setting Up and Executing Tasks from Research Repositories) targets ML and NLP tasks taken from research repositories. Agents must set up a repository and run a task, for example downloading the OpenBookQA dataset and counting examples in its train, dev, and test splits. The task data is hosted on the Hugging Face Hub in three sets: Expert (45 problems), Masked (152), and AutoGen (602).
Running and Evaluating Agents
After cloning the repo and installing requirements, users add an OpenAI API key to a .env file. A single query can be run with the run_single_query module and an environment backend. The local backend executes code on the user's machine, which the README flags as a risk. Docker and modal.com are offered as alternatives, and modal.com is the backend used for the benchmark evaluation. The run_on_benchmark module evaluates the implemented agents on a chosen set, such as Expert. Agent trajectories from the paper's experiments are included in the repository.
Related Resources
The README links to the arXiv paper, a Hugging Face leaderboard, and the Hugging Face dataset.
Community Discussions
Be the first to start a conversation about SUPER Benchmark
Share your experience with SUPER Benchmark, ask questions, or help others learn from your insights.
Pricing
Open Source
Self-hosted SUPER benchmark code and resources under the Apache License 2.0. External costs (OpenAI API, optional Modal compute) are not charged by the project.
- License: Apache License 2.0 (Apache-2.0)
- Benchmark sets: Expert (45 problems), Masked (152), AutoGen (602)
- Run agent locally, in Docker, or on modal.com
- Requires your own OpenAI API key (external cost)
- Modal compute is an external cost; roughly 2-3 cents per average problem, with $30 credit per month per the README
Capabilities
Key Features
- Benchmark tasks for setting up and executing ML/NLP research repositories
- Expert, Masked and AutoGen task sets
- Reference agent runnable via a single-query command
- Local, Docker and modal.com execution backends
- Benchmark evaluation script
- Agent trajectories from paper experiments
- Public leaderboard and Hugging Face dataset