# QbitOS

> A 6B parameter sparse MoE language model delivering large-model reasoning quality at small-model compute cost, deployable on consumer hardware with as little as 4GB VRAM.

QbitOS is a 6B parameter language model engineered by QbitOS Labs for efficient deployment on consumer-grade hardware. It combines Sparse Mixture-of-Experts (MoE) with Grouped Query Attention (GQA) and advanced quantization techniques to deliver performance the project claims is comparable to 13B dense models while requiring only 4GB VRAM or 8GB system RAM. The repository was created in July 2026 and is available under an Educational/Research License on GitHub.

## What It Is

QbitOS is a foundation model and inference stack designed to bring frontier-level reasoning, coding, and agent capabilities to resource-constrained environments. The core model has 6B total parameters but activates only 2B per token via sparse MoE routing, making it significantly cheaper to run than equivalently capable dense models. It targets a full pipeline from pretraining through quantized edge deployment, and includes built-in support for tool use, function calling, and multimodal input via a 150M vision encoder.

## Architecture and Design

QbitOS is built around several efficiency-focused architectural choices:

- **Sparse MoE**: 8 experts with top-2 routing per token, plus a shared expert that is always active, giving 6B total capacity at 2B active compute
- **Grouped Query Attention (GQA)**: 32 query heads with 8 key/value heads, reducing KV-cache memory by 4×
- **32K context window** with RoPE position encoding extended via YaRN scaling
- **SwiGLU activation** and RMSNorm (pre-LN) for training stability
- **Knowledge distillation** from 70B teacher models to improve reasoning at small scale
- **AWQ + KV INT8 quantization** for sub-4GB VRAM deployment

The multimodal extension fuses a 150M ViT encoder with the transformer stack via a cross-modal projection layer.

## Deployment Model

QbitOS ships a complete production stack alongside the model weights and training code:

- **Docker and Docker Compose** for local inference and monitoring
- **Kubernetes manifests** including HPA for horizontal scaling
- **OpenAI-compatible API server** for drop-in compatibility
- **vLLM integration** with PagedAttention for high-throughput serving
- **Prometheus + Grafana + Jaeger** observability stack

The README lists inference targets ranging from edge devices (8GB RAM, CPU-only, GGUF Q4_0) to high-end production servers (16GB+ VRAM, FP16/BF16). On an RTX 4090, the project reports 120 tokens/second throughput; on a CPU with 8 cores, approximately 8 tokens/second.

## Performance Claims

The README publishes benchmark scores the project attributes to its own evaluation suite:

- **MMLU**: 68–72%, described as comparable to 13B dense models
- **GSM8K**: 72–76%, described as comparable to 70B models with Chain-of-Thought
- **HumanEval**: 58–62%, described as comparable to CodeLlama-7B
- **BBH**: 62–65%, described as comparable to 13B dense models

These figures are vendor-published and have not been independently verified.

## Safety and Alignment

QbitOS includes a dedicated safety layer with Constitutional AI training via DPO, real-time content filtering, post-generation output moderation, an automated red-teaming suite, and sandboxed tool execution. The repository includes a `safety/` module with separate files for content filtering, constitutional AI, red teaming, and moderation.

## Current Status

The repository was created on July 7, 2026, and last updated the same day, making it a very early-stage public release at version 1.0. It carries an Educational/Research License that permits academic and personal use but restricts commercial deployment without explicit written permission. The project has 1 star and 0 forks as of its initial publication.

## Features
- Sparse Mixture-of-Experts (MoE) architecture with 6B total / 2B active parameters
- Grouped Query Attention (GQA) with 4× KV-cache reduction
- 32K token context window with RoPE + YaRN scaling
- Knowledge distillation from 70B teacher models
- AWQ + KV INT8 quantization for 4GB VRAM deployment
- Speculative decoding for up to 2.5× inference speedup
- OpenAI-compatible API server
- vLLM integration with PagedAttention
- Multimodal support via 150M vision encoder
- Tool use and function calling with structured output
- Constitutional AI safety and content filtering
- Docker, Kubernetes, and CI/CD production stack
- Prometheus + Grafana + Jaeger observability
- GGUF export for CPU/edge deployment
- Pretraining, SFT, DPO, and distillation training pipeline

## Integrations
vLLM, Docker, Kubernetes, Prometheus, Grafana, Jaeger, WandB, llama.cpp, FlashAttention, CUDA

## Platforms
WINDOWS, MACOS, LINUX, WEB, API, CLI

## Pricing
Open Source

## Version
1.0

## Links
- Website: https://github.com/QbitOSLabs/QbitOS
- Repository: https://github.com/QbitOSLabs/QbitOS
- EveryDev.ai: https://www.everydev.ai/tools/qbitos