WebVoyager

WebVoyager is the code and data release accompanying the paper "WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models." It implements a web agent powered by large multimodal models that completes user instructions by interacting with real-world websites through a Selenium-based browser environment. The repository is published under the Apache 2.0 license, and its disclaimer states it is not an officially supported Tencent product.

What It Is

WebVoyager is a research implementation of a web agent plus its evaluation tooling. The agent combines textual and visual information, using screenshots with overlaid bounding boxes on interactive elements, to decide on navigation actions. A text-only mode uses the page accessibility tree instead of screenshots. The repo also ships a task dataset and an automated evaluator.

How the Workflow Runs

Users set up a conda environment with Chrome installed, copy tasks into a test file, add an OpenAI API key to run.sh, and launch the agent. Parameters control the maximum number of interaction iterations, the number of recent screenshots kept in context, temperature, seed, headless browsing, window size, and the model used. Each task produces a results folder containing interaction messages and screenshots. Prompts are defined in prompts.py and the action format can be modified in run.py.

Data and Evaluation

The dataset has 643 task queries covering 15 websites, each with 40+ queries, plus reference answers. It also includes 90 web browsing tasks extracted from the GAIA validation set, starting from Google Search. Some tasks, mainly on Booking and Google Flights, are time-sensitive and need their dates updated manually. Results can be judged by human review of screenshots and agent actions, or by an automated GPT-4V protocol that evaluates the task, the agent's response, and the last screenshots. The README also provides a sample prompt for expanding the task pool with GPT-4.

Limitations to Know

The disclaimer notes that outputs are affected by non-deterministic OpenAI API behavior, prompt changes, and website style changes or pop-ups, and the project does not guarantee accuracy.