LLMLingua

LLMLingua is an open-source Python library developed by Microsoft Research that compresses prompts for large language models (LLMs) by up to 20x, reducing inference costs and latency with minimal performance degradation. It uses a compact, well-trained language model (e.g., GPT2-small, LLaMA-7B) to identify and remove non-essential tokens from prompts. The library includes three main methods — LLMLingua, LongLLMLingua, and LLMLingua-2 — each targeting different compression scenarios, plus SecurityLingua for jailbreak defense.

• LLMLingua compresses prompts using a small language model to identify and drop low-importance tokens; install via pip install llmlingua and use the PromptCompressor class to compress any prompt.
• LongLLMLingua addresses the "lost in the middle" problem in long-context LLMs, improving RAG performance by up to 21.4% using only 1/4 of the tokens; use the rank_method="longllmlingua" parameter.
• LLMLingua-2 is a task-agnostic compression method trained via data distillation from GPT-4, offering 3x–6x faster performance than LLMLingua; enable it with use_llmlingua2=True.
• SecurityLingua is a safety guardrail that uses security-aware prompt compression to detect jailbreak attacks with 100x fewer token costs than state-of-the-art guardrail approaches.
• Structured Prompt Compression allows fine-grained control over which sections to compress using <llmlingua></llmlingua> tags with optional rate and compress parameters.
• Cost Savings are achieved by reducing both prompt and generation lengths, with reported savings on GPT-4 API usage.
• KV-Cache Compression accelerates the inference process by compressing the key-value cache.
• Framework Integrations include LangChain, LlamaIndex, and Microsoft Prompt Flow, making it easy to drop into existing RAG pipelines.
• No LLM Retraining Required — the compression is applied at inference time without modifying the target LLM.
• Quantized Model Support allows running with models like TheBloke/Llama-2-7b-Chat-GPTQ using under 8GB of GPU memory.