# Kimi Vendor Verifier

> Open-source toolkit from Moonshot AI for verifying third-party inference providers' Kimi model deployments via benchmarks and API conformance tests.

Kimi Vendor Verifier is an open-source evaluation toolkit from Moonshot AI for checking how accurately third-party providers serve Kimi models. It combines inspect-ai benchmarks, pytest API conformance checks, a long-memory benchmark script and an agent benchmark, and the README publishes K3 results for several providers.

## What It Is

The project is a Python test harness that a vendor or evaluator points at a Kimi model endpoint by setting an API key and base URL. It runs pre-flight validation first, then scored benchmarks, so deployment differences across providers can be compared against Moonshot's own results.

## Pre-flight API Checks

Pytest suites verify raw chat-completions behavior:
- `tests/params/` checks that immutable parameters such as temperature, top_p, presence_penalty, frequency_penalty and n are constrained correctly.
- `tests/tool_call_json_schema/` sends walle-valid schemas as tool parameters and validates returned tool-call arguments locally, in both streaming and non-streaming modes.
- `tests/k3_features/` validates K3 features including dynamic tools, response_format, tool_choice and thinking effort.
- `tests/prompt_tokens/` compares vendor-reported `usage.prompt_tokens` with expected constants for text and vision cases.

## Benchmarks

The inspect-ai tasks are OCRBench, MMMU Pro Vision and AIME 2025 (no longer used for K3). BEAM (1M) is a standalone long-term memory benchmark over 1M-token conversations. DeepSWE, a set of 113 coding-agent tasks, runs on the Pier platform using Kimi-code 0.23.6 or later. The CLI supports thinking mode, K3 reasoning effort (low, high, max), streaming, concurrency control, automatic retry on network errors, and resuming interrupted runs.

## Setup Path

Install with `uv sync && uv pip install -e .`, set KIMI_API_KEY and KIMI_BASE_URL, then run the pytest checks followed by `eval.py` for the benchmarks. The README recommends running OCRBench first as a quick deployment sanity check. It is released under the MIT License.

## Features
- OCRBench and MMMU Pro Vision evaluations via inspect-ai
- BEAM (1M) long-term memory benchmark
- DeepSWE coding-agent benchmark on the Pier platform
- API parameter constraint validation
- Tool-call JSON schema validation (streaming and non-streaming)
- K3 feature contract tests
- Prompt token usage accuracy checks
- Thinking mode and reasoning effort configuration
- Automatic retry with exponential backoff
- Provider comparison results table for K3

## Integrations
inspect-ai, pytest, vLLM, SGLang, KTransformers, Pier, DeepSWE, Kimi-code

## Platforms
API, CLI

## Pricing
Open Source

## Links
- Website: https://github.com/MoonshotAI/Kimi-Vendor-Verifier
- Documentation: https://github.com/MoonshotAI/Kimi-Vendor-Verifier
- Repository: https://github.com/MoonshotAI/Kimi-Vendor-Verifier
- EveryDev.ai: https://www.everydev.ai/tools/kimi-vendor-verifier