Jeeves

Jeeves is an open-source project from PostHog that trains a Jev-style decision classifier on Qwen3.5-9B so it reasons before it decides. It accepts a state plus yes/no, multiple-choice and rating questions, and returns calibrated probabilities. The repository includes the training code, train/dev/test data builders, inference server, a Python SDK and a diffusion drafter for faster reasoning.

What It Is

Jeeves is a decision model and serving stack. It uses Qwen3.5-9B with LoRA adapters and a pointer head that scores each answer option. The model produces a reasoning chain, then scores the options with a softmax whose temperature is fitted on a dev set. Requests use Jev's format: noul for yes/no, choice for multiple-choice and score for ratings, all in the same request.

Training and Reported Results

Training has two stages: supervised fine-tuning, then CISPO reinforcement learning, followed by temperature calibration. The README reports 0.889 accuracy on its test data against 0.822 for Kev-9B and 0.857 for Jev. On the 231 public JevBench items it reports 0.935 against 0.866 for Jev. These are the authors' own figures. The README also notes that Jeeves trails Jev on knowledge-heavy questions such as MMLU and MMLU-Pro.

Serving and Speed Options

The server is started with python -m inference.serve and runs on CUDA in bf16 or FP8. Inference on Apple Silicon (MPS) is also supported. A block-4 diffusion drafter speeds up reasoning, and the README reports about 1.6x on one question. Request options are:

• think
• max_think
• nothink_threshold
• return_reasoning

These trade latency against accuracy. The README reports about 0.3 s per request without thinking and a 3.3 s median with full thinking on one H100.

Python SDK and Reproduction

The sdk/ folder provides jeeves_sdk, a drop-in replacement for Jev's Python SDK. It connects to a local server by default and needs no API key. Scripts for building the data, training, evaluation, JevBench, export and FP8 quantization are included. Weights are published on Hugging Face.

Limitations

The README states that thinking has a slow tail, with 17 s at p90 on one H100 with full chains. It also states that no language-consistency reward was used, so reasoning chains are not well interpretable.