NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is open source inference serving software from NVIDIA that lets teams deploy AI models from frameworks such as TensorRT, PyTorch, ONNX, OpenVINO, Python and RAPIDS FIL. The current release documented is 2.73.0, and the project is licensed under BSD-3-Clause. NVIDIA states it is part of NVIDIA AI Enterprise.

What It Is

Triton is a model serving layer that loads models from a file-system model repository and exposes them for inference over HTTP/REST, gRPC, or an in-process C API. Each model has its own scheduler, which can batch requests and hand them to the backend matching the model type. Triton supports NVIDIA GPUs, x86 and ARM CPUs, and AWS Inferentia across cloud, data center, edge and embedded devices.

How Serving Works

Models are placed in a repository, optionally with a model configuration, and served from the NGC Triton container. Clients such as the Python and C++ libraries send requests, and the documented quickstart runs an ONNX image classification model and returns the top predicted classes. Features include concurrent model execution, dynamic and sequence batching, implicit state management for stateful models, and model pipelines via ensembles or Business Logic Scripting. A model management API loads and unloads models at runtime.

Extensibility and Operations

A backend API allows custom backends and pre/post-processing in C/C++ or Python. Repository agents can add functionality at model load and unload time. Metrics report GPU utilization, throughput and latency, and health endpoints help integration with Kubernetes. Performance Analyzer and Model Analyzer help tune configurations, and documentation covers TensorRT-LLM and vLLM backends, OpenAI-compatible API, and deployment examples for Kubernetes on AWS and GCP.