VeloxML

VeloxML is an open-source CLI and deployment engine that lets data scientists and developers deploy open-source large language models directly into their own AWS or GCP cloud accounts with a single command. Licensed under Apache 2.0, it is built on top of SkyPilot — an open-source compute orchestrator developed at UC Berkeley — and is currently in active alpha development.

What It Is

VeloxML sits in the MLOps and LLM infrastructure category. Its core job is to eliminate the operational overhead of deploying private LLM inference endpoints: no Dockerfiles to write, no Kubernetes YAML to manage, and no proprietary framework decorators to adopt. Users define a model service in a plain app.py and a minimal veloxml.yaml hardware spec, then run veloxml deploy to get a live, curl-testable endpoint running on a GPU instance inside their own VPC.

How the Deployment Workflow Works

The CLI follows a three-step pattern:

• Init: veloxml init my-model scaffolds a project directory with app.py and veloxml.yaml.
• Deploy: veloxml deploy reads the app code and hardware spec, provisions an optimized Spot or On-Demand GPU instance via SkyPilot, installs dependencies, verifies a /health probe, and prints a ready-to-use endpoint URL.
• Teardown: veloxml down --all terminates all cloud compute to prevent lingering charges.

The tool defaults to Spot instances (use_spot: true) to reduce GPU costs. The README notes that a real open-weights model (Qwen2.5-0.5B-Instruct) can be deployed to AWS on a Spot instance in under two minutes.

Architecture and Stack

VeloxML is built in Python and leverages two major open-source projects:

• SkyPilot (UC Berkeley) for compute orchestration and multi-cloud instance provisioning across AWS and GCP.
• Truss (Baseten) for model packaging and serving conventions.

Supported serving frameworks include vLLM, Hugging Face Transformers, FastAPI, and pure Python. The tool targets any open-weights checkpoint or fine-tuned model, and the README lists use cases including private LLM inference APIs, custom embedding and reranking microservices, agent tool-calling backends, domain-specific fine-tuned models, and high-throughput batch inference endpoints.

Data Sovereignty and Compliance Angle

A central design principle of VeloxML is that all data, prompts, and model weights remain inside the user's own cloud account. The README explicitly positions this against managed inference platforms (Modal, RunPod, Baseten), arguing that VeloxML avoids third-party server exposure and eliminates the need for SOC2 or HIPAA compliance reviews with external vendors. The tool also avoids framework lock-in by not requiring proprietary decorators like those used by Modal.

Update: v0.1.1 — Rolling Updates & DX Improvements

The latest release is v0.1.1, published on September 4, 2026, titled "Rolling Updates & DX Improvements." The repository was created in late August 2026 and last updated in early September 2026. The project self-describes its current status as Alpha — under heavy development — with Beta and 1.0 milestones still ahead. The README advises watching releases for a Beta notification and notes that breaking changes should be expected.