Rilldex RL Studio
Local RL observability workspace for LLM agents: behavior maps, trainer metrics from TRL, verl and more, run comparisons, reward hacking checks, and Python/SQL access.
At a Glance
Rilldex is free to install from its public GitHub repository, with no paid tier. The repository has no license file, so it is not formally open source. Semantic maps and model-based reward checks call the Jev/Typesafe API with your own key, plus any hosted LLM you choose, billed by those providers.
Engagement
Available On
Listed Nov 2026
About Rilldex RL Studio
Rilldex RL Studio is a Python-based reinforcement learning observability tool by Shelter Zhou for teams training LLM agents. It collects rollouts and native trainer metrics, maps what the agent did and how its actions relate, and links every measurement back to its source evidence. You can work through a local dashboard, Python, SQL or a mix, before, during or after training.
What It Is
Rilldex is a single-user workspace that runs on your machine and stores everything in a live SQLite database. It takes in conversations, native tool histories and environment episodes from files, Python or HTTP, including OpenAI-style and Anthropic tool calls, native Harbor and SWE-agent directories, and saved Weave call exports. From that data it builds behavior maps of actions, feedback, plans, branches and retries, and connects them to training signals such as reward, loss, entropy, KL, clipping, tokens and throughput.
Training Telemetry and Run Comparison
Trainer telemetry plugs into TRL, ms-swift GRPO and Stable-Baselines3 through callbacks, imports verl logs, or accepts custom metric definitions for other trainers and inference servers. You can align up to three runs on a shared axis, select an interval and open the learner updates and rollouts behind it. Run capture records console logs, configuration, Python and Git context, CPU, RAM and NVIDIA GPU samples where available, and versioned file snapshots. An execution timeline shows worker spans, failures and weight-sync events, and a policy mismatch view measures token probability gaps from log probabilities you already have, with no extra inference.
Behavior and Revision Analysis
The rollout story shows each action's purpose, changes of method and returns to earlier work, grouped by agent context. Revision analysis shows what triggered a change, what was changed and what followed, with gains and regressions side by side. Behavior comparisons contrast activities and follow-up between training windows and save the exact cases behind them, and a context scan shows an agent step's input mixture and repeated blocks. Investigations, pinned views and workspace search keep saved evidence easy to reopen.
Reward Hacking Checks
Built-in checks scan rollouts for answer acquisition, evaluator access, evaluation manipulation, conflicts with task instructions, reward inconsistencies and artifact integrity problems, and can follow new rollouts during training. You can write custom reward detectors in Python, validate detector versions against frozen review sets, and share offline findings reports. The README states that flags indicate suspected behavior rather than proven exploitation and that no general real-world accuracy rate is established.
Model Setup
Collection, standard telemetry, source maps and analysis need no LLM. Semantic maps and the behavior-based reward checks use a decision model through the Jev/Typesafe API, set with a JEV_API_KEY, or a custom decision backend you implement; semantic maps also need a candidate LLM, which can be a local model such as one served by Ollama or a hosted chat-completions API. Call and byte budgets cap model use, and saved decisions are reused on reruns.
Setup
Install with pip from the GitHub repository on Python 3.11 or later, on Windows, Linux or macOS, with no GPU needed for collection, then run rilldex start --open. A bundled example needs no API key. A Docker image can be built from the repository, optional extras add cloud storage readers for S3 and Google Cloud Storage, and remote jobs connect through SSH forwarding. The repository does not include a license file.
Community Discussions
Be the first to start a conversation about Rilldex RL Studio
Share your experience with Rilldex RL Studio, ask questions, or help others learn from your insights.
Pricing
Free
Rilldex is free to install from its public GitHub repository, with no paid tier. The repository has no license file, so it is not formally open source. Semantic maps and model-based reward checks call the Jev/Typesafe API with your own key, plus any hosted LLM you choose, billed by those providers.
- Local dashboard, CLI and Python API
- Collection, trainer telemetry and SQL analysis with no LLM required
- Bundled example that needs no API key
- Docker build from the repository
Capabilities
Key Features
- Behavior maps of agent actions, feedback, plans, branches and retries
- Rollout story with action purposes and changes of method
- Revision analysis with gains and regressions together
- Behavior comparisons between training windows
- Run comparisons aligning up to three runs
- Native trainer telemetry for TRL, ms-swift GRPO and Stable-Baselines3
- verl log import and custom metric definitions
- Run capture of console logs, config, Python/Git context, hardware and file snapshots
- Execution timeline of worker spans, failures and weight-sync events
- Policy mismatch measurement from existing log probabilities
- Context scan of an agent step's inputs
- Built-in reward hacking checks
- Custom reward detectors in Python
- Detector validation sets and offline findings reports
- Threshold notifications and Python-defined monitors
- Continuous collection with durable outboxes and resume
- Live SQLite workspace queryable from Python and SQL
- JSONL/CSV export
- Weave calls export import
- Long-context source reader with your chosen model
- Model call and byte budgets with reused decisions
- Docker deployment and SSH forwarding for remote jobs
- Optional S3 and Google Cloud Storage readers
