# VeloxML

> Open-source CLI tool that deploys open-source LLMs to your own AWS or GCP account with a single command, with no Docker, no Kubernetes, and automatic scale-to-zero.

VeloxML is an open-source CLI and deployment engine that lets data scientists and developers deploy open-source large language models directly into their own AWS or GCP cloud accounts with a single command. Licensed under Apache 2.0, it is built on top of SkyPilot — an open-source compute orchestrator developed at UC Berkeley — and is currently in active alpha development.

## What It Is

VeloxML sits in the MLOps and LLM infrastructure category. Its core job is to eliminate the operational overhead of deploying private LLM inference endpoints: no Dockerfiles to write, no Kubernetes YAML to manage, and no proprietary framework decorators to adopt. Users define a model service in a plain `app.py` and a minimal `veloxml.yaml` hardware spec, then run `veloxml deploy` to get a live, curl-testable endpoint running on a GPU instance inside their own VPC.

## How the Deployment Workflow Works

The CLI follows a three-step pattern:

- **Init**: `veloxml init my-model` scaffolds a project directory with `app.py` and `veloxml.yaml`.
- **Deploy**: `veloxml deploy` reads the app code and hardware spec, provisions an optimized Spot or On-Demand GPU instance via SkyPilot, installs dependencies, verifies a `/health` probe, and prints a ready-to-use endpoint URL.
- **Teardown**: `veloxml down --all` terminates all cloud compute to prevent lingering charges.

The tool defaults to Spot instances (`use_spot: true`) to reduce GPU costs. The README notes that a real open-weights model (Qwen2.5-0.5B-Instruct) can be deployed to AWS on a Spot instance in under two minutes.

## Architecture and Stack

VeloxML is built in Python and leverages two major open-source projects:

- **SkyPilot** (UC Berkeley) for compute orchestration and multi-cloud instance provisioning across AWS and GCP.
- **Truss** (Baseten) for model packaging and serving conventions.

Supported serving frameworks include vLLM, Hugging Face Transformers, FastAPI, and pure Python. The tool targets any open-weights checkpoint or fine-tuned model, and the README lists use cases including private LLM inference APIs, custom embedding and reranking microservices, agent tool-calling backends, domain-specific fine-tuned models, and high-throughput batch inference endpoints.

## Data Sovereignty and Compliance Angle

A central design principle of VeloxML is that all data, prompts, and model weights remain inside the user's own cloud account. The README explicitly positions this against managed inference platforms (Modal, RunPod, Baseten), arguing that VeloxML avoids third-party server exposure and eliminates the need for SOC2 or HIPAA compliance reviews with external vendors. The tool also avoids framework lock-in by not requiring proprietary decorators like those used by Modal.

## Update: v0.1.1 — Rolling Updates & DX Improvements

The latest release is **v0.1.1**, published on September 4, 2026, titled "Rolling Updates & DX Improvements." The repository was created in late August 2026 and last updated in early September 2026. The project self-describes its current status as **Alpha** — under heavy development — with Beta and 1.0 milestones still ahead. The README advises watching releases for a Beta notification and notes that breaking changes should be expected.

## Features
- One-command LLM deployment to AWS or GCP
- No Docker or Kubernetes required
- Automatic scale-to-zero
- Spot instance support for cost efficiency
- Data sovereignty — model weights and prompts stay in your own VPC
- Supports vLLM, Hugging Face, FastAPI, and pure Python
- Built on SkyPilot compute orchestrator
- YAML-based hardware spec configuration
- Health probe verification before endpoint goes live
- Teardown command to avoid lingering cloud charges

## Integrations
AWS, GCP, SkyPilot, vLLM, Hugging Face Transformers, FastAPI, Truss (Baseten), Qwen, uvicorn

## Platforms
CLI, API

## Pricing
Open Source

## Version
v0.1.1

## Links
- Website: https://www.veloxml.com/
- Documentation: https://github.com/paguasmar/veloxml-deploy#getting-started
- Repository: https://github.com/paguasmar/veloxml-deploy
- EveryDev.ai: https://www.everydev.ai/tools/veloxml