tokensift

tokensift is a token-efficiency linter for LLM prompts and payloads, built by Riten Vagadiya and published under the MIT license. It performs deterministic, local, tokenizer-level static analysis on prompt strings, Message[] arrays, and tool schemas — no network calls, no accounts, no telemetry. The project is actively developed and available on npm as tokensift.

What It Is

tokensift works like a code linter (think ESLint), but instead of flagging style issues, it flags token waste: text that costs money and context-window space without adding value for the model. It actually tokenizes input using the same encoder the provider uses — for OpenAI models that means the real BPE vocabulary, so counts are exact. For Claude, where no public tokenizer exists, it uses a calibrated estimate and marks every finding with confidence: "estimate" vs "exact". The tool ships 20 built-in rules covering problems like UUID bloat, repeated blocks, pretty-printed JSON, base64 blobs, filler phrases, and more.

How the Analysis Works

Each finding includes the rule that fired, the token cost before and after a fix, a suggestion, and a real dollar cost derived from a curated snapshot of LiteLLM's pricing table. Finding.cost.perCall is the savings per call; Finding.cost.per1000Calls is the same number at a more readable denomination. A volume config option projects monthly cost at a given request rate. The analyze() function runs all built-in rules by default; passing rules: [...] runs a subset, and rules: [] just tokenizes with no findings.

Key built-in rules include:

• uuid-bloat — UUIDs cost ~18 tokens; a short id costs 3
• repeated-block — verbatim spans repeated across a prompt are paid every time
• pretty-json — indented JSON spends tokens on whitespace the model doesn't need
• base64-blob — base64 runs close to 1 token per 1.3–1.5 characters (error severity)
• filler — hedging phrases add token cost with no instruction content
• html-whitespace, whitespace-run, unicode-punct — all autofixable

Two Ways to Use It

As a library: analyze() accepts a prompt string or message array and returns a Report with findings, summary token counts, and cost. dyn() marks dynamic slots so static and per-request token costs are tracked separately. defineRule lets you write custom rules in the same shape as the 20 built-ins. Test matchers for vitest and jest are available via tokensift/matchers, with toBeUnderTokens, toHaveNoTokensiftErrors, and toMatchTokenBaseline.

As a CLI: Point it at files, globs, or stdin. Output formats include pretty terminal output, JSON, GitHub Actions annotations (--format github), PR-comment markdown tables (--format markdown), and SARIF 2.1.0 for GitHub Code Scanning. --fix --write applies safe autofixes. tokensift init scaffolds a config file and reference snippets for GitHub Actions, pre-commit hooks, and test setup. tokensift check is the CI gate: exits 0 (clean) or 2 (error-severity finding), with no middle ground.

Deployment and Runtime Compatibility

The analyze/budget/tokenize path has no Node-specific code, so it works in Cloudflare Workers, Supabase Edge Functions, Netlify Edge Functions, and Deno via npm:tokensift — no config, no shims. The full bundle gzips to ~1.6MB; importing a single encoder family (e.g., tokensift/encoders/o200k) drops it to ~1.13MB, useful for edge environments with compressed-size limits. Vercel users should use the Node.js runtime rather than Edge Runtime.

Update: v1.3.0

The latest release is v1.3.0, published on 2026-08-28. The project was created in July 2026 and has been actively developed since, with the core engine, 20 rules, CLI, test matchers, Claude calibration support, and pricing tooling all functional. Supported OpenAI model families are o200k_base (gpt-4o, gpt-4.1, gpt-5, o1, o3, o4-mini, and more) and cl100k_base (gpt-4, gpt-3.5-turbo), both with exact confidence. Anthropic support covers claude-opus-4-5, claude-sonnet-4-5, and claude-haiku-4-5 with estimate-based confidence and a calibrate command for running your own calibration against your own prompts and API key.