# S-Agent 2.0

> A spatial reasoning agent that pairs the OpenHands coding agent with a command-line 3D perception toolkit for multi-view and video questions.

S-Agent 2.0 is a research system from Ropedia that gives a general coding agent (OpenHands) a command-line perception toolkit for spatial reasoning over images and video. The agent acquires 3D scene evidence, keeps it in persistent scene memory, and reads measurements back in plain spatial language. The repository contains the task instruction, perception gateway, benchmark loaders, agent harness and GPU perception server; model weights are not included.

## What It Is

S-Agent 2.0 is an agentic spatial reasoning framework rather than a trained model. It needs no training and no custom controller: the toolkit is a single executable (`perception.py`) the agent calls from its shell, and rendered images are files the agent opens with its ordinary file viewer. The paper and project page present it as closing an "evidence loop": acquire, remember, interpret.

## How the Toolkit Works

Tool perception commands build the scene: `recon` creates a metric, gravity-aligned reconstruction of all frames, and `segment` lifts SAM 3 text-prompted masks into named 3D instances. Model perception commands such as `add_instance` let the agent register a point or an oriented observer implied by the question, and `drop` removes errors. Readout commands (`list_instance`, `bev`, `distance`, `depth`, `side`, `elevation`, `motion`, `compass`) return JSON with metric values, units, and a conclusion sentence that states the reference frame. Frozen models behind the toolkit include Depth Anything 3, Pi3MOS, GeoCalib and SAM 3.

## Evaluation and Results

The project page reports results on MMSI-Bench, ReVSI, MindCube, SPAR-Bench and MMSI-Video with Qwen3.8-27B and Gemini-3.8-Flash backbones. It states 74.1 on MMSI-Bench and 90.5 on ReVSI with Gemini-3.8-Flash, and compares against SpatialClaw, pySpatial and SpaceTools. Ablations and runs with Codex and Claude Code as the coding agent are also reported, along with eleven browsable successful trajectories.

## Setup and Limitations

Setup involves four services plus a task runner: a GPU perception server, a gateway, an LLM endpoint for the backbone, and the OpenHands runtime image, with a `scripts/run.sh` launcher for benchmarks. Documented limitations: grounding depends on the backbone, which may misplace an imagined observer or confuse object height with height above the floor, and reconstruction errors in scale or pose propagate to readouts. Code under agent/ and docs/ is Apache 2.0, while the perception component keeps NVIDIA's non-commercial license.

## Features
- Command-line 3D perception toolkit callable from a coding agent's shell
- Metric, gravity-aligned scene reconstruction (recon)
- Text-prompted SAM 3 segmentation into named 3D instances
- Persistent scene memory of frames, geometry and instances
- Oriented observer placement via add_instance
- Spatial readouts: distance, depth, side, elevation, motion, compass, bev
- Readouts returned as JSON with units and reference-frame conclusions
- Benchmark loaders and evaluation harness for MMSI-Bench, ReVSI, SPAR, MindCube and MMSI-Video
- Replaceable perception models via the CLI
- Works with OpenHands, Codex and Claude Code as coding agents

## Integrations
OpenHands, SpatialClaw, Depth Anything 3, SAM 3, GeoCalib, Pi3MOS, LiteLLM, vLLM, Codex, Claude Code

## Platforms
API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Links
- Website: https://ropedia.github.io/S-Agent-2.0
- Documentation: https://github.com/Ropedia/S-Agent-2.0/blob/main/docs/SETUP.md
- Repository: https://github.com/Ropedia/S-Agent-2.0
- EveryDev.ai: https://www.everydev.ai/tools/s-agent-2-0