Kimi Vendor Verifier

Kimi Vendor Verifier is an open-source evaluation toolkit from Moonshot AI for checking how accurately third-party providers serve Kimi models. It combines inspect-ai benchmarks, pytest API conformance checks, a long-memory benchmark script and an agent benchmark, and the README publishes K3 results for several providers.

What It Is

The project is a Python test harness that a vendor or evaluator points at a Kimi model endpoint by setting an API key and base URL. It runs pre-flight validation first, then scored benchmarks, so deployment differences across providers can be compared against Moonshot's own results.

Pre-flight API Checks

Pytest suites verify raw chat-completions behavior:

• tests/params/ checks that immutable parameters such as temperature, top_p, presence_penalty, frequency_penalty and n are constrained correctly.
• tests/tool_call_json_schema/ sends walle-valid schemas as tool parameters and validates returned tool-call arguments locally, in both streaming and non-streaming modes.
• tests/k3_features/ validates K3 features including dynamic tools, response_format, tool_choice and thinking effort.
• tests/prompt_tokens/ compares vendor-reported usage.prompt_tokens with expected constants for text and vision cases.

Benchmarks

The inspect-ai tasks are OCRBench, MMMU Pro Vision and AIME 2025 (no longer used for K3). BEAM (1M) is a standalone long-term memory benchmark over 1M-token conversations. DeepSWE, a set of 113 coding-agent tasks, runs on the Pier platform using Kimi-code 0.23.6 or later. The CLI supports thinking mode, K3 reasoning effort (low, high, max), streaming, concurrency control, automatic retry on network errors, and resuming interrupted runs.

Setup Path

Install with uv sync && uv pip install -e ., set KIMI_API_KEY and KIMI_BASE_URL, then run the pytest checks followed by eval.py for the benchmarks. The README recommends running OCRBench first as a quick deployment sanity check. It is released under the MIT License.