ort

ort is an open-source Rust interface from pykeio for running hardware-accelerated inference and training on machine learning models in the ONNX format. It is primarily a wrapper around Microsoft's ONNX Runtime and also supports other pure-Rust backends. The documentation describes version 2.0.0-rc.14 as production-ready.

What It Is

ort is a Rust crate that lets developers deploy models exported from PyTorch, TensorFlow, Keras, scikit-learn, or PaddlePaddle by loading the resulting ONNX graph and running it through ONNX Runtime. It targets both on-device and datacenter deployment. The project is Apache-2.0 licensed.

Workflow

Add the crate to Cargo.toml, convert your model to ONNX with the framework's exporter, then create an Environment and a Session. The session builder can set the graph optimization level, thread counts, and the model file. Prepare inputs, call run(), and extract output tensors as arrays.

Performance and Backends

Execution providers enable hardware acceleration, including NVIDIA CUDA and TensorRT, Intel OpenVINO, Qualcomm QNN, and Huawei CANN. IoBinding controls where inputs and outputs are placed to improve I/O efficiency. Alternative backends include ort-candle, ort-tract, and ort-web for browser deployment.

Ecosystem Use

The README lists open-source projects that use ort, including Text Embeddings Inference, Magika, FastEmbed-rs, and Ultralytics YOLO Rust Inference. The docs also cover platform support, Cargo features, linking, prebuilt binaries, and version mapping.