jeff

jeff is a self-hosted implementation of TypeSafe's jev System One API, built by Logan Markewich and released under the MIT license. It runs on the GLiFormer large model (400M parameters) and is designed to be a cost-effective alternative to the hosted jev service, compatible with the official typesafe-sdk by pointing the TYPESAFE_BASE_URL environment variable at a jeff instance.

What It Is

jeff is an open-source classification inference server that replicates the TypeSafe jev System One API surface. It answers structured classification questions about a given text state, supporting three question types: choice (pick one option from a set), score (rate on ordered levels), and noul (probability of yes/no). The server exposes a REST API compatible with the official TypeSafe SDK, so existing SDK-based applications can switch to jeff with only an environment variable change.

How It Works

jeff wraps the GLiFormer encoder model in a batching HTTP server built with Python. Incoming requests are queued and batched for efficient GPU throughput. The server auto-selects the compute device in order: CUDA → MPS → CPU. Key architectural details include:

• Batching: Configurable batch size and wait time (JEFF_MAX_BATCH, JEFF_MAX_WAIT_MS) for throughput tuning.
• Isolation modes: Nouls get separate encoder passes by default; choice and score questions share a pass. JEFF_ISOLATE=all forces full independence at extra compute cost.
• ONNX backend: An optional ONNX Runtime path runs the encoder for CPU deployments, with int8 quantization support.
• Rate limiting and auth: Per-key bearer token auth and per-key rate limiting are configurable via environment variables.

Deployment Options

jeff supports local development and cloud deployment on Modal:

• Local: Requires uv and Python 3.12. Run with uv run jeff after syncing dependencies and downloading model weights from Hugging Face.
• Modal GPU: Deploy with an L4 GPU for the HTTP API. The README reports measured HTTP throughput capping at approximately 50 requests/second per container, with horizontal scaling via JEFF_MAX_CONTAINERS.
• Modal CPU / ONNX: Available as a fallback, though the README notes that an 8-core Modal CPU deployment was slower and more expensive than the hosted jev service.
• Mac: MPS is preferred over CPU for local development.

Benchmarks and Tradeoffs

The README includes benchmark results measured on 1,600 labeled items across eight datasets, comparing jeff to the hosted jev service:

• jeff achieves approximately 75.5% accuracy on AG News topic classification vs. 90.5% for jev.
• On the JevBench v1.2.2 leaderboard (18 models), jeff ranks #9 overall but #14 of 18 on intelligence score alone; its overall rank is driven by cost efficiency.
• jeff is described as close to jev on binary sentiment and emotion classification, but substantially behind on irony, reading comprehension, and harder reasoning tiers.
• The README estimates cost per 1M single-question requests at approximately $2.6 (L4/Modal HTTP) vs. approximately $15.6 for jev — roughly a 6× cost reduction at the expense of accuracy on reasoning-heavy tasks.

Current Status

The repository was created in September 2026 and last updated shortly after, with 243 stars and 18 forks at the time of indexing. The project is actively maintained under the MIT license and accepts contributions via standard GitHub workflow. Tests are included, with model integration tests requiring a local model checkpoint and SDK live tests using a fake backend server.