# tokensift

> A deterministic, offline token-efficiency linter for LLM prompts and payloads that identifies token waste and calculates real dollar costs per finding.

tokensift is a token-efficiency linter for LLM prompts and payloads, built by Riten Vagadiya and published under the MIT license. It performs deterministic, local, tokenizer-level static analysis on prompt strings, `Message[]` arrays, and tool schemas — no network calls, no accounts, no telemetry. The project is actively developed and available on npm as `tokensift`.

## What It Is

tokensift works like a code linter (think ESLint), but instead of flagging style issues, it flags token waste: text that costs money and context-window space without adding value for the model. It actually tokenizes input using the same encoder the provider uses — for OpenAI models that means the real BPE vocabulary, so counts are exact. For Claude, where no public tokenizer exists, it uses a calibrated estimate and marks every finding with `confidence: "estimate"` vs `"exact"`. The tool ships 20 built-in rules covering problems like UUID bloat, repeated blocks, pretty-printed JSON, base64 blobs, filler phrases, and more.

## How the Analysis Works

Each finding includes the rule that fired, the token cost before and after a fix, a suggestion, and a real dollar cost derived from a curated snapshot of LiteLLM's pricing table. `Finding.cost.perCall` is the savings per call; `Finding.cost.per1000Calls` is the same number at a more readable denomination. A `volume` config option projects monthly cost at a given request rate. The `analyze()` function runs all built-in rules by default; passing `rules: [...]` runs a subset, and `rules: []` just tokenizes with no findings.

Key built-in rules include:
- **uuid-bloat** — UUIDs cost ~18 tokens; a short id costs 3
- **repeated-block** — verbatim spans repeated across a prompt are paid every time
- **pretty-json** — indented JSON spends tokens on whitespace the model doesn't need
- **base64-blob** — base64 runs close to 1 token per 1.3–1.5 characters (error severity)
- **filler** — hedging phrases add token cost with no instruction content
- **html-whitespace**, **whitespace-run**, **unicode-punct** — all autofixable

## Two Ways to Use It

**As a library**: `analyze()` accepts a prompt string or message array and returns a `Report` with findings, summary token counts, and cost. `dyn()` marks dynamic slots so static and per-request token costs are tracked separately. `defineRule` lets you write custom rules in the same shape as the 20 built-ins. Test matchers for vitest and jest are available via `tokensift/matchers`, with `toBeUnderTokens`, `toHaveNoTokensiftErrors`, and `toMatchTokenBaseline`.

**As a CLI**: Point it at files, globs, or stdin. Output formats include pretty terminal output, JSON, GitHub Actions annotations (`--format github`), PR-comment markdown tables (`--format markdown`), and SARIF 2.1.0 for GitHub Code Scanning. `--fix --write` applies safe autofixes. `tokensift init` scaffolds a config file and reference snippets for GitHub Actions, pre-commit hooks, and test setup. `tokensift check` is the CI gate: exits `0` (clean) or `2` (error-severity finding), with no middle ground.

## Deployment and Runtime Compatibility

The `analyze`/`budget`/`tokenize` path has no Node-specific code, so it works in Cloudflare Workers, Supabase Edge Functions, Netlify Edge Functions, and Deno via `npm:tokensift` — no config, no shims. The full bundle gzips to ~1.6MB; importing a single encoder family (e.g., `tokensift/encoders/o200k`) drops it to ~1.13MB, useful for edge environments with compressed-size limits. Vercel users should use the Node.js runtime rather than Edge Runtime.

## Update: v1.3.0

The latest release is v1.3.0, published on 2026-08-28. The project was created in July 2026 and has been actively developed since, with the core engine, 20 rules, CLI, test matchers, Claude calibration support, and pricing tooling all functional. Supported OpenAI model families are `o200k_base` (gpt-4o, gpt-4.1, gpt-5, o1, o3, o4-mini, and more) and `cl100k_base` (gpt-4, gpt-3.5-turbo), both with exact confidence. Anthropic support covers `claude-opus-4-5`, `claude-sonnet-4-5`, and `claude-haiku-4-5` with estimate-based confidence and a `calibrate` command for running your own calibration against your own prompts and API key.

## Features
- Token-efficiency linting for LLM prompts and payloads
- 20 built-in rules covering UUID bloat, repeated blocks, pretty-printed JSON, base64 blobs, filler phrases, and more
- Exact tokenization for OpenAI models using real BPE vocabulary
- Calibrated estimate-based tokenization for Claude models
- Real dollar cost per finding with per-call and per-1000-calls breakdown
- CLI with file, glob, and stdin support
- Output formats: terminal, JSON, GitHub Actions annotations, Markdown, SARIF 2.1.0
- Autofixable rules: unicode-punct, whitespace-run, pretty-json, html-whitespace
- Custom rule authoring via defineRule
- Test matchers for vitest and jest (toBeUnderTokens, toHaveNoTokensiftErrors, toMatchTokenBaseline)
- Template slot tracking with dyn() for static vs dynamic token cost separation
- Baseline regression tracking for CI
- Token budget enforcement with budget init and check commands
- Pricing snapshot from LiteLLM with local override support
- Cloudflare Workers, Supabase Edge, Netlify Edge, and Deno compatible
- No telemetry, no accounts, no background network calls
- Deterministic and fully offline analysis

## Integrations
OpenAI (gpt-4o, gpt-4.1, gpt-5, o1, o3, o4-mini, and more), Anthropic Claude (claude-opus-4-5, claude-sonnet-4-5, claude-haiku-4-5), vitest, jest, GitHub Actions, Cloudflare Workers, Supabase Edge Functions, Netlify Edge Functions, Vercel (Node.js runtime), Deno, LiteLLM pricing table

## Platforms
WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v1.3.0

## Links
- Website: https://github.com/ritenv/tokensift
- Documentation: https://github.com/ritenv/tokensift/blob/main/DESIGN.md
- Repository: https://github.com/ritenv/tokensift
- EveryDev.ai: https://www.everydev.ai/tools/tokensift