# NVIDIA Triton Inference Server

> Open source inference serving software that deploys AI models from multiple frameworks across cloud, data center, edge and embedded devices.

NVIDIA Triton Inference Server is open source inference serving software from NVIDIA that lets teams deploy AI models from frameworks such as TensorRT, PyTorch, ONNX, OpenVINO, Python and RAPIDS FIL. The current release documented is 2.73.0, and the project is licensed under BSD-3-Clause. NVIDIA states it is part of NVIDIA AI Enterprise.

## What It Is

Triton is a model serving layer that loads models from a file-system model repository and exposes them for inference over HTTP/REST, gRPC, or an in-process C API. Each model has its own scheduler, which can batch requests and hand them to the backend matching the model type. Triton supports NVIDIA GPUs, x86 and ARM CPUs, and AWS Inferentia across cloud, data center, edge and embedded devices.

## How Serving Works

Models are placed in a repository, optionally with a model configuration, and served from the NGC Triton container. Clients such as the Python and C++ libraries send requests, and the documented quickstart runs an ONNX image classification model and returns the top predicted classes. Features include concurrent model execution, dynamic and sequence batching, implicit state management for stateful models, and model pipelines via ensembles or Business Logic Scripting. A model management API loads and unloads models at runtime.

## Extensibility and Operations

A backend API allows custom backends and pre/post-processing in C/C++ or Python. Repository agents can add functionality at model load and unload time. Metrics report GPU utilization, throughput and latency, and health endpoints help integration with Kubernetes. Performance Analyzer and Model Analyzer help tune configurations, and documentation covers TensorRT-LLM and vLLM backends, OpenAI-compatible API, and deployment examples for Kubernetes on AWS and GCP.

## Features
- Multiple deep learning and machine learning framework backends
- Concurrent model execution
- Dynamic batching
- Sequence batching and implicit state management
- Model ensembles and Business Logic Scripting
- HTTP/REST and gRPC protocols based on KServe
- C and Java in-process APIs
- Custom backends in C/C++ and Python
- Model management API for loading and unloading models
- Metrics for GPU utilization, throughput and latency
- Performance Analyzer and Model Analyzer tools
- TensorRT-LLM and vLLM backends

## Integrations
TensorRT, PyTorch, ONNX Runtime, OpenVINO, RAPIDS FIL, vLLM, TensorRT-LLM, Kubernetes, KServe, AWS Inferentia, NVIDIA AI Enterprise, Docker

## Platforms
LINUX, WEB, API, CLI

## Pricing
Open Source

## Version
2.73.0

## Links
- Website: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
- Documentation: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
- Repository: https://github.com/triton-inference-server/server
- EveryDev.ai: https://www.everydev.ai/tools/nvidia-triton-inference-server