AI Tools &amp; Discussions in Local Inference

TensorRT LLM

LLM Inference Optimization for NVIDIA GPUs

ExLlamaV3

Local LLM Inference Library

AI Dev Libraries Local Inference AI Infrastructure

ort

Rust ONNX Runtime Inference Library

ExLlamaV2

Local LLM Inference Library for GPUs

Shimmy

Local GGUF Inference Server Rust

Local Inference AI Infrastructure LLM Orchestration

vllm-mlx

Local LLM Server for Apple Silicon

AWQ (llm-awq)

LLM Weight Quantization Toolkit

Model Management Local Inference Compute Opt.

LoRAX

LoRA Adapter Inference Server

Local Inference AI Infrastructure Compute Opt.

AI Tools & Discussions in Local Inference

Tools and platforms for running AI inference locally without cloud dependence.

Local Inference Tools (269)

KTransformers

CPU GPU LLM Inference Framework

vLLM Ascend

vLLM Plugin for Huawei Ascend

TensorRT LLM

LLM Inference Optimization for NVIDIA GPUs

ExLlamaV3

Local LLM Inference Library

AI Dev Libraries Local Inference AI Infrastructure

ort

Rust ONNX Runtime Inference Library

ExLlamaV2

Local LLM Inference Library for GPUs

Shimmy

Local GGUF Inference Server Rust

Local Inference AI Infrastructure LLM Orchestration

vllm-mlx

Local LLM Server for Apple Silicon

AWQ (llm-awq)

LLM Weight Quantization Toolkit

Model Management Local Inference Compute Opt.

LoRAX

LoRA Adapter Inference Server