# SUPER Benchmark

> A benchmark and agent code for evaluating LLM agents on setting up and executing ML/NLP tasks from research repositories.

SUPER is a benchmark from allenai for evaluating LLM agents on setting up and executing tasks from research repositories on GitHub. The repository provides the benchmark resources along with agent code that can be run locally, in Docker, or on modal.com, and it is released under the Apache License 2.0.

## What It Is

SUPER (Evaluating Agents on Setting Up and Executing Tasks from Research Repositories) targets ML and NLP tasks taken from research repositories. Agents must set up a repository and run a task, for example downloading the OpenBookQA dataset and counting examples in its train, dev, and test splits. The task data is hosted on the Hugging Face Hub in three sets: Expert (45 problems), Masked (152), and AutoGen (602).

## Running and Evaluating Agents

After cloning the repo and installing requirements, users add an OpenAI API key to a .env file. A single query can be run with the run_single_query module and an environment backend. The local backend executes code on the user's machine, which the README flags as a risk. Docker and modal.com are offered as alternatives, and modal.com is the backend used for the benchmark evaluation. The run_on_benchmark module evaluates the implemented agents on a chosen set, such as Expert. Agent trajectories from the paper's experiments are included in the repository.

## Related Resources

The README links to the arXiv paper, a Hugging Face leaderboard, and the Hugging Face dataset.

## Features
- Benchmark tasks for setting up and executing ML/NLP research repositories
- Expert, Masked and AutoGen task sets
- Reference agent runnable via a single-query command
- Local, Docker and modal.com execution backends
- Benchmark evaluation script
- Agent trajectories from paper experiments
- Public leaderboard and Hugging Face dataset

## Integrations
Hugging Face, Modal, OpenAI, Docker

## Platforms
API, CLI

## Pricing
Open Source

## Links
- Website: https://github.com/allenai/super-benchmark
- Repository: https://github.com/allenai/super-benchmark
- EveryDev.ai: https://www.everydev.ai/tools/super-benchmark