ClawGUI

ClawGUI is an open-source research framework from the ZJU-REAL organization that covers the lifecycle of GUI agents: online reinforcement learning training, standardized evaluation, and deployment on real phones. It is released under the Apache 2.0 license and described in an arXiv paper. The repository is organized into independent modules, each with its own environment and documentation.

What It Is

ClawGUI addresses three coupled problems in building GUI agents: an environment to train the agent online, benchmarks to measure what it learned, and a system to run it on real devices. Each module covers one of these. The framework also includes ClawGUI-2B, a 2B agent trained end-to-end with the pipeline. The README reports 17.1 MobileWorld success rate versus an 11.1 baseline.

The Modules

• ClawGUI-RL trains agents with online RL. It runs many Docker-based Android emulators in parallel or trains on physical or cloud Android devices. It uses GiGPO with a process reward model for step-level rewards, and includes spare-server rotation and episode visualization.
• ClawGUI-Eval uses an Infer, Judge, Metric pipeline across 6 benchmarks (ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl) and 11+ models. It supports local GPU or OpenAI-compatible API backends. The README reports a 95.8% reproduction rate against official results.
• ClawGUI-Agent controls Android (ADB), HarmonyOS (HDC), and iOS (XCTest) devices by natural language from 12+ chat platforms. It supports several GUI models through OpenAI-compatible APIs, personalized memory, episode recording, a Gradio web UI, and one-sentence benchmark runs.
• ClawGUI-Skills implements training-free self-evolving skills. Skills are stored as structured packages, retrieved at run time, and revised after failed runs are diagnosed. Modes are off, trace, reuse, and evolve.
• ClawGUI-APP runs the brain LLM and phone agent on a single Android phone via Shizuku, with no desktop coordinator. It offers voice input, session memory, and trace replay.

Setup Path

Clone the repository and open the module you need; each has separate installation instructions. The roadmap lists planned desktop and web online RL, more agent skills, and a hybrid CLI and GUI mechanism. For ClawGUI-APP, the brain and VLM are still served via cloud API today.