# ClawGUI

> Open-source framework for training GUI agents with online RL, evaluating them on standardized benchmarks, and deploying them on real devices.

ClawGUI is an open-source research framework from the ZJU-REAL organization that covers the lifecycle of GUI agents: online reinforcement learning training, standardized evaluation, and deployment on real phones. It is released under the Apache 2.0 license and described in an arXiv paper. The repository is organized into independent modules, each with its own environment and documentation.

## What It Is

ClawGUI addresses three coupled problems in building GUI agents: an environment to train the agent online, benchmarks to measure what it learned, and a system to run it on real devices. Each module covers one of these. The framework also includes ClawGUI-2B, a 2B agent trained end-to-end with the pipeline. The README reports 17.1 MobileWorld success rate versus an 11.1 baseline.

## The Modules

- **ClawGUI-RL** trains agents with online RL. It runs many Docker-based Android emulators in parallel or trains on physical or cloud Android devices. It uses GiGPO with a process reward model for step-level rewards, and includes spare-server rotation and episode visualization.
- **ClawGUI-Eval** uses an Infer, Judge, Metric pipeline across 6 benchmarks (ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl) and 11+ models. It supports local GPU or OpenAI-compatible API backends. The README reports a 95.8% reproduction rate against official results.
- **ClawGUI-Agent** controls Android (ADB), HarmonyOS (HDC), and iOS (XCTest) devices by natural language from 12+ chat platforms. It supports several GUI models through OpenAI-compatible APIs, personalized memory, episode recording, a Gradio web UI, and one-sentence benchmark runs.
- **ClawGUI-Skills** implements training-free self-evolving skills. Skills are stored as structured packages, retrieved at run time, and revised after failed runs are diagnosed. Modes are off, trace, reuse, and evolve.
- **ClawGUI-APP** runs the brain LLM and phone agent on a single Android phone via Shizuku, with no desktop coordinator. It offers voice input, session memory, and trace replay.

## Setup Path

Clone the repository and open the module you need; each has separate installation instructions. The roadmap lists planned desktop and web online RL, more agent skills, and a hybrid CLI and GUI mechanism. For ClawGUI-APP, the brain and VLM are still served via cloud API today.

## Features
- Online RL training with parallel Docker Android environments
- Real-device RL training on physical or cloud Android phones
- GiGPO with PRM step-level rewards
- Standardized evaluation on 6 GUI benchmarks
- Natural-language control of Android, HarmonyOS, and iOS devices
- Self-evolving skills with failure diagnosis and revision
- Personalized memory and episode recording
- Gradio web UI
- On-device Android deployment via Shizuku
- Voice input via OpenAI-compatible speech-to-text

## Integrations
Docker, Android ADB, HarmonyOS HDC, iOS XCTest, Shizuku, OpenAI-compatible APIs, Hugging Face, ModelScope, Feishu, Qwen-VL, UI-TARS, AutoGLM, MAI-UI, GUI-Owl

## Platforms
LINUX, ANDROID, IOS, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
clawgui-app-v0.3.1

## Links
- Website: https://github.com/ZJU-REAL/ClawGUI
- Documentation: https://github.com/ZJU-REAL/ClawGUI#readme
- Repository: https://github.com/ZJU-REAL/ClawGUI
- EveryDev.ai: https://www.everydev.ai/tools/clawgui