EveryDev.ai
Subscribe
Home
Tools

5,559+ AI tools

  • Featured
  • Gems
  • Trending
  • New
  • Rate tools
  • Compare
  • Arena
Categories
  • Agents2807
  • Coding2096
  • Infrastructure997
  • Projects781
  • MCP708
  • Research702
  • Analytics535
  • Marketing516
  • Design459
  • Testing448
  • Data420
  • Security380
  • Integration277
  • Communication271
  • Prompts265
  • Extensions231
  • Voice193
  • DevOps184
  • Commerce180
  • Web125
  • Learning116
  • Finance41
AI Tools by Topic
  • AI Coding Assistants
  • Agent Frameworks
  • MCP Servers
  • AI Prompt Tools
  • Vibe Coding Tools
  • AI Design Tools
  • AI Database Tools
  • AI Website Builders
  • AI Testing Tools
  • LLM Evaluations
Follow Us
  • X / Twitter
  • LinkedIn
  • Reddit
  • Discord
  • Threads
  • Bluesky
  • Mastodon
  • YouTube
  • GitHub
  • Instagram
Get Started
  • Users
  • Rate Tools
  • About
  • Editorial Standards
  • Corrections & Disclosures
  • Community Guidelines
  • Advertise
  • Contact Us
  • Newsletter
  • Submit a Tool
  • Start a Discussion
  • Write A Blog
  • Share A Build
  • Terms of Service
  • Privacy Policy
Explore with AI
  • ChatGPT
  • Gemini
  • Claude
  • Grok
  • Perplexity
Agent Experience
  • llms.txt
Theme
With AI, Everyone is a Dev. EveryDev.ai ©
    1. Home
    2. Tools
    3. switch
    switch icon

    switch

    LLM Evaluations

    A TypeScript framework for building decision-model harnesses that give a model an agent loop over finite choices.

    Visit Website

    At a Glance

    Pricing
    Open Source

    switch framework released under the MIT License; free to use, copy, modify and distribute. Model inference via third-party providers is an external cost.

    Engagement

    Available On

    API
    SDK
    CLI

    Resources

    WebsiteGitHubllms.txt

    Topics

    LLM EvaluationsAgent HarnessAI Decision Models

    Alternatives

    jevcoreDrex DLMAnyJev
    Developer
    Onur SolmazOnur Solmaz builds open-source developer tooling, with pi-wo…

    Listed Nov 2026

    About switch

    switch is an open-source TypeScript framework by osolmaz for building harnesses around decision models. Your code turns the state of a task into typed questions, such as which next step to take from a short list, and the model picks the answers. The latest listed release is v0.1.1.

    What It Is

    switch is a harness framework for decision models like TypeSafe's Jev. A decision model answers typed questions with probabilities and writes no text, so it cannot generate the tool calls that a coding agent harness relies on. switch generates the tool calls in code, and the model only picks one from that finite set. You implement a domain, which defines the state of a task and the functions that turn a state into the model's questions and its answers into the next state. Chess is a domain, and so is a rubric judge for evals.

    How a Run Works

    The domain makes the initial state, and the question function gives the questions for that state. The model reads a view of the state and answers every question. The transition function then makes the next state from the answers. The loop ends when there are no more questions, when a request fails, or at a step limit the domain may set. Questions can be bool, choice or score types, and a picking helper builds a choice question from a list of choices.

    Traces, Replay and Examples

    Every run writes a trace of states, questions and answers. A replay reads that trace without calling a model, and can fork from a given step so a model continues from there. The run, view and replay commands come from a Commander-based command line helper, and a browser viewer plays runs back step by step. The repository includes a rubric judge that navigates documents to grade rubric items, a chess example, and a WikiRace example.

    Setup Path

    The package installs from npm and requires Node.js 22.18 or newer. It works with any classifier model that pi-ai lists, and separate entry points cover the core loop, model access, the command line and testing with a fake decider. The examples and viewer are in the repository and run from a pnpm workspace.

    switch - 1

    Community Discussions

    Be the first to start a conversation about switch

    Share your experience with switch, ask questions, or help others learn from your insights.

    Pricing

    OPEN SOURCE

    Open Source (MIT)

    switch framework released under the MIT License; free to use, copy, modify and distribute. Model inference via third-party providers is an external cost.

    • MIT License
    • Install from npm: @osolmaz/switch
    • Requires Node.js 22.18 or newer
    • Domain, harness, run, replay and trace tooling
    • Examples and viewer live in the repository and are not on npm

    Capabilities

    Key Features

    • Agent loop for finite-choice decision models
    • Typed bool, choice and score questions
    • Domain state machines with question, transition and view functions
    • Picking helper for choice questions
    • Trace recording and model-free replay
    • Replay forking from a chosen step
    • Run, view and replay command line via Commander
    • Browser viewer for runs
    • Fake decider for tests without model calls
    • Example domains: rubric judge, chess, WikiRace

    Integrations

    pi-ai
    TypeSafe Jev
    llama.cpp
    Commander
    Stockfish
    API Available

    Demo Video

    switch Demo Video
    Watch on YouTube

    Ratings & Reviews

    No ratings yet

    Be the first to rate switch and help others make informed decisions.

    Rate other tools you’ve used

    Developer

    Onur Solmaz

    Onur Solmaz builds open-source developer tooling, with pi-workflows as the primary public project. The project extends the Pi coding agent with a durable TypeScript-based workflow engine, terminal viewers, and a resource manager subsystem. The work is released under the MIT license and distributed via npm and crates.io.

    Read more about Onur Solmaz
    WebsiteGitHub
    2 tools in directory

    Similar Tools

    jevcore icon

    jevcore

    TypeSafe Jev integration for DeepSeek Harness and MCP hosts that delivers typed judgments (yes/no, choice, score) instead of prose, offline by default with explicit egress disclosure.

    Drex DLM icon

    Drex DLM

    A decision model from Nace.AI that returns probabilities for typed questions about a given document using an 8B diffusion language model with a pointer head.

    AnyJev icon

    AnyJev

    Open-source Python library that turns an open LLM into a Jev-style decision model returning typed decisions with calibrated probabilities.

    Browse all tools

    Related Topics

    LLM Evaluations

    Platforms and frameworks for evaluating, testing, and benchmarking LLM systems and AI applications. These tools provide evaluators and evaluation models to score AI outputs, measure hallucinations, assess RAG quality, detect failures, and optimize model performance. Features include automated testing with LLM-as-a-judge metrics, component-level evaluation with tracing, regression testing in CI/CD pipelines, custom evaluator creation, dataset curation, and real-time monitoring of production systems. Teams use these solutions to validate prompt effectiveness, compare models side-by-side, ensure answer correctness and relevance, identify bias and toxicity, prevent PII leakage, and continuously improve AI product quality through experiments, benchmarks, and performance analytics.

    232 tools

    Agent Harness

    Infrastructure, orchestrators, and task runners that wrap around LLM coding agents — covering session management, context delivery, worktree isolation, architecture enforcement, and issue-to-PR pipelines.

    400 tools

    AI Decision Models

    Models that make decisions instead of generating text. Given context and predefined options, they return choices, scores, or probabilities that applications can act on.

    19 tools
    Browse all topics
    Back to all toolsSuggest an edit
    ratings
    discussions
    3views