TensorRT LLM

TensorRT LLM is an open-source library from NVIDIA for optimizing LLM and Visual Generation model inference on NVIDIA GPUs. It combines custom kernels for common operations, runtime optimizations, and a PyTorch-based Python LLM API. The documentation site lists 1.4.0rc0 as the current docs version, and the repository README shows release 1.4.0rc1.

What It Is

TensorRT LLM is an inference optimization and serving library. The README says it provides custom kernels for operations such as attention, GEMMs, and MoE, plus runtime optimizations including prefill-decode disaggregation, Wide Expert Parallelism, and speculative decoding. Its high-level Python LLM API supports setups from a single GPU to multi-GPU and multi-node deployments. Its PyTorch-native architecture is designed to be modular and easy to modify, and several popular models are predefined and can be customized with native PyTorch code.

Workflow and Tooling

The documentation describes several ways to run models:

• Online serving with trtllm-serve, with client examples for OpenAI-style chat, completion, and responses endpoints, plus multimodal and LoRA examples.
• Offline inference through the LLM API, and through a VisualGen API for visual generation.
• trtllm-bench for benchmarking and trtllm-eval for evaluation.
• Slurm examples for multi-node runs.
• Installation through pip, container images, or building from source.

The README states that the LLM API integrates with the wider inference ecosystem, including NVIDIA Dynamo and Triton Inference Server.

Performance and Serving Features

Documented features include in-flight batching with paged attention, KV cache reuse, compression, offloading and connectors, quantization, speculative decoding, guided decoding, LoRA, sparse attention, long-sequence support, and multiple parallelism strategies such as Helix parallelism. The docs also cover disaggregated serving, sub-agent routing, embeddings for encoder-only models, multimodal support, and a Ray orchestrator marked as a prototype. Model recipes cover models such as Nemotron, GPT-OSS, and Qwen3. Visual generation support is labeled Beta in the docs.

Telemetry and Licensing

The README says anonymous telemetry is collected by default and can be disabled with environment variables, a file, a Python API setting, or a CLI flag. The project is licensed under Apache 2.0, with some bundled third-party components under other licenses.