LOFT

LOFT (Long Context Frontiers) is an open-source benchmark from Google DeepMind, introduced in the paper "Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?". It groups 35 datasets into 6 long-context task categories spanning retrieval, multi-hop compositional reasoning and more, across 4 modalities. The repository provides dataset downloads, prompt conversion, inference and evaluation code.

What It Is

LOFT is a benchmark suite for testing whether long-context language models can handle tasks usually done by dedicated systems. Its tasks include text, visual and audio retrieval, RAG, SQL, and many-shot in-context learning. Datasets come in context lengths such as 32k, 128k and 1m, with corpus, dev, few-shot and test query files for each.

Workflow and Outputs

Users install the Python requirements, then run a download script that fetches the datasets under a chosen directory. Some datasets are preprocessed to infill missing fields in queries and corpus files. The repository includes an example Corpus-in-Context (CiC) prompt for text retrieval. Inference currently supports Gemini models (for example gemini-1.5-flash-002) through Vertex AI and requires a Google Cloud project ID. An evaluation script then scores predictions, with each task type producing multiple metrics such as recall@1, mrecall, subspan exact match and SQL execution accuracy. A prompt-dumping script exports prompts as text or CSV for third-party evaluation.

LOFT-Hard Subset

The authors found that Gemini 1.5 already performed well on many LOFT datasets but showed headroom on others. They recommend iterating on MuSiQue, QAMPARI and QUEST, and provide prompts for those three.

Licensing and Data Notes

Software is licensed under Apache 2.0 and other materials under CC-BY 4.0. Individual tasks may carry their own licensing. For Flickr30k, MS COCO and MSR-VTT, users must download the datasets from their original websites. The repository states it is not an official Google product.