Preference Model
Preference Model is a superintelligence data research company building robust reinforcement-learning environments for training capable, better-aligned AI systems. It focuses on reward functions, secure harnesses and graders, and environments for AI research and ML engineering.
At a Glance
- Frontier AI research labs
- AI model developers and post-training teams
- ML research and engineering organizations
- Teams building agent benchmarks and evaluation environments
AI Tools by Preference Model
(1)
Karotte
Open Source RL Environment Framework
Discussions
No discussions yet
Be the first to start a discussion about Preference Model
Latest News
Published 'Building RL Environments for Superintelligence,' outlining the company's approach to robust environments for increasingly capable models.
Emerged from stealth and announced a $16 million seed round led by a16z.
Open-sourced Karotte, its framework for building robust RL environments.
Andreessen Horowitz published its investment announcement describing Preference Model's focus on AI research and ML-engineering environments.
Products & Services
An open-source framework for building robust reinforcement-learning environments. Developers write tasks in Python; Karotte runs agents in sandboxes, provides tools and graders, scores outcomes, and records transcripts. It uses secure defaults including unprivileged execution, resource limits, firewalling, process cleanup, and protected grading artifacts.
Research and development of RL environments and associated training tasks for frontier AI labs, especially environments for ML research, software engineering, GPU kernels, data curation, post-training, and computer-use-style work.
Market Position
Preference Model positions itself as a specialist in robust, adversarially tested RL environments for AI research and ML engineering, rather than static labeling datasets. Its differentiation is security-minded infrastructure designed for models that actively optimize against graders, with defenses hardened through more than one million evaluation runs. Relevant alternatives include labs' in-house environment teams and RL/evaluation frameworks such as Harbor, HUD, AgentEnv, verifiers, Inspect, Habitat, DeepTune, Fleet, Vmax, Turing, Mechanize, and Bespoke.
Leadership
Founders
Jennifer Zhou
Co-founder and CEO. Previously worked on Anthropic's data team, building data infrastructure, tokenizers, and datasets, and earlier worked at Stripe.
Ning Cao
Co-founder, leading strategy, business development, and recruiting. Previously an early employee at DatologyAI, where he helped build the company from 0 to 1.
Executive Team
Jennifer Zhou
Co-founder and Chief Executive Officer
Former Anthropic data-team member who worked on data infrastructure, tokenizers, and Claude pretraining datasets; previously worked at Stripe.
Ning Cao
Co-founder; Strategy, Business Development, and Recruiting
Early DatologyAI employee who helped build that company from 0 to 1.
Founding Story
The founders started Preference Model around the view that alignment and capability depend heavily on the quality of the reward signals used in training. After seeing data and training infrastructure up close at Anthropic and DatologyAI, they set out to build robust, secure RL environments that prevent reward hacking and help frontier models learn useful behavior rather than loopholes.
Business Model
Revenue Model
The company builds and sells custom RL environments and training data to frontier AI labs; its Karotte framework is open source and functions as a public framework around the company's environment-development work.
Target Markets
- Frontier AI research labs
- AI model developers and post-training teams
- ML research and engineering organizations
- Teams building agent benchmarks and evaluation environments
- Training and evaluating AI models on ML research and engineering tasks
- Writing and optimizing GPU/CUDA kernels
- Debugging training runs and designing experiments
- Post-training and reasoning-token efficiency
- Data curation and labeling
- Software engineering and coding environments