Jev Cookbook

Jev Cookbook is a community-maintained, MIT-licensed GitHub repository by chr-kelly that provides copy-paste-ready question sets ("recipes") for TypeSafe's Jev structured-decision API. It ships alongside an evaluation harness with reproducible CLINC150 benchmark results and a linter that catches silent API mis-reads before they reach production.

What It Is

Jev is a typed, low-latency API that answers structured questions about a piece of text — classifying intent, scoring urgency, or flagging attributes — without generating free-form prose. The Jev Cookbook fills the gap between the API's three primitives (choice, score, noul) and real-world usage: it shows practitioners what to ask and why each question is shaped that way, including what was tried first, what broke, and where each recipe still gets things wrong.

Recipes Included

Each recipe lives in its own folder with a questions.json ready to paste into a request and a README explaining the design decisions:

• customer-support-routing — category, refund intent, repeat contact, urgency
• roleplay-state — tone drift, stalled tension, lore breaks as live sensors
• agent-tool-guardrail — whether an agent's tool call matches its own stated plan
• context-pruning — which old tool calls in a long transcript still earn their place
• llm-router — which model tier should handle a request, judged on the work not the model
• content-qa — whether a generated page is thin, and which of five ways

Four Design Rules

The cookbook enforces four rules that exist because skipping any one of them produces wrong answers on real data:

• Every choice has a fallback option — Jev picks from a closed set and will choose confidently even on off-topic input; always include OTHER/UNRESOLVED and review what lands there.
• Ask atomic questions, combine in code — separate urgency, refund intent, and repeat-contact into individual questions, then weight them with your own formula so changing priorities means editing a coefficient, not rewriting a prompt.
• Multi-label means several nouls, not one choice — questions in one call are evaluated independently and in parallel, so a dozen nouls cost roughly what one does.
• Split judgement from extraction — Jev generates nothing; use it to gate, then pass survivors to an LLM only where prose is needed.

The Two-Pass Pattern

The cookbook documents a two-pass architecture for large corpora: a cheap gate of 5–7 questions runs on everything, roughly 10% survive to a full 40–60 question checklist, and an LLM is invoked only on that filtered set. Because questions inside a single call are nearly free but state is what you pay for, this is where the order-of-magnitude cost savings come from.

Eval Harness and Benchmarks

The eval/ directory contains a harness that runs any recipe against a labelled dataset and reports agreement, a confidence-vs-error curve, escalation rate at a given threshold, and fallback-option behavior. The README publishes one measured run: CLINC150 test set plus 1,000 out-of-scope items, one 151-way choice, untuned option names, run on 2026-09-21 against jev-1.13.0. The run reported 89.4% agreement, 90.6% in-scope accuracy, 83.8% OOS recall, and an ECE of 0.025 at approximately $0.63 total cost. Full per-item results and the exact command are included in the repo. The project explicitly states: "No numbers are published here that you cannot reproduce."

Provider Compatibility

The same request body works against TypeSafe's own endpoint (api.typesafe.ai), OpenRouter (openrouter.ai/api/v1/decisions), and NanoGPT (nano-gpt.com/api/v1/decisions). The --base-url flag on the eval harness points it at any compatible server. At the time of the README, TypeSafe's direct API was gated behind a waitlist; the other two providers were open.