VoiceGremlin

VoiceGremlin is a SaaS tool founded in 2026 by Nick Hall that automates end-to-end testing of intelligent voice agents that answer phones. A test run places a real call to the agent's phone number, and the call transcript is evaluated by a judge model that returns a pass/fail result. It is driven entirely by a REST API, with no SDK.

What It Is

VoiceGremlin is a testing service for voice agents (IVAs). Each test is a plain-English string describing what to validate, and a test suite is an array of strings, so there is no test case repository or scripted conversation path to maintain. One customized agent calls the target and role-plays as a customer. A second agent reads the transcript and judges whether the target met the goal, using inference rather than keyword matching or regex.

How the Workflow Runs

A CI job makes two calls. First, POST /api/runs with a phone number and a list of test goals (1 to 50, with optional concurrency and a custom run_id). Then poll GET /api/runs/:run_id every 5 to 10 seconds until the status moves from queued to running to complete. Results include a boolean passed value per test, a reason, latency, and the full call transcript, so failures can be diagnosed directly. Calls go out over a real carrier line, which exercises the SIP integration, STT/TTS and the agent's behavior. Supported numbers are US/Canada.

CI Validation Targets

Special no-charge phone numbers let teams check their pipeline without placing real calls: instant pass, slow pass (180s), always-fail and flaky targets. Two simulated agents, an airline reservation agent and a hotel front desk agent, run LLM-to-LLM conversations without deducting minutes.

Limits to Know

The about page says adversarial or red-team testing is possible but the tool is not optimized for it and should not be relied on. There are no webhooks or callbacks, so results must be polled. The service is intended only for calling voice agents, and cold calls or marketing use can lead to account suspension.