# FluidUse

> Swift package for local computer use on Apple silicon that fills forms and makes typed decisions with on-device Core ML models via the macOS Accessibility API.

FluidUse is an Apache-2.0 Swift package from FluidInference for local computer use on Apple silicon. It reads a form in a running Mac app or browser through the Accessibility API, asks a small on-device model what belongs in each field, and types the answer into the real app. The README states that nothing leaves the machine and that a decision takes about 1 ms on the Neural Engine. The latest release listed is v0.3.0.

## What It Is

FluidUse is a Swift library with demo apps and a CLI for running small Core ML decision models on a Mac. Its first model is CUA-S1-FORMS, a 706K-parameter form specialist from Cua, converted to Core ML and served by FluidAudio. Beyond form filling, it exposes managers for typed decision models that answer choice, yes/no and score questions about a piece of text, and in some cases images, with probabilities.

## How Form Filling Works

An AccessibilityFormDriver attaches to any running application and takes a snapshot of its fields, labels and values. The app builds a set of options from a profile, such as fill, check, click or skip, and the model scores them for each actionable field. The decoded decision is then typed or clicked through the driver. WebFormDriver does the same for an embedded WKWebView. DocumentEntities builds the profile from a PDF or text file of Label: value lines, and an optional PredeterminedAnswer sheet covers question-style fields the model does not decide.

## Supported Model Runtimes

The README documents managers for several model families:
- laya: a 322M encoder with a decision head, running on the Neural Engine.
- GLiNER 2.5: classification heads in base and multilingual variants.
- Kev-0.8B and Intern-Decision-0.8B: text decision models running on the GPU.
- Decision 2.0 and Decision 3.0: vLLM Semantic Router models, with image input for the d3 models.
- Clef-vision: an image decision model.
- Vela 2.0: routing, safety and PII span detection.
- ShortReply: drafts short replies to social posts for human review.
- Unlimited-OCR: multi-page document OCR that outputs markdown with layout boxes.

Models download from Hugging Face on first use, and the README publishes latency and accuracy comparisons against PyTorch references in Benchmarks.md.

## Scope and Setup

The package is added through Swift Package Manager. Running the demo requires Accessibility access for the launching terminal. The README says the model matches a field to a value it is given. It does not read résumés, reason about dropdown options or write free text, and uploads and essays are left to the person. Submit is never clicked unless it is enabled. Some models list macOS 14 or 15 and iOS 17 or 18 minimums.

## Features
- Form filling in Mac apps and browsers via the Accessibility API
- On-device Core ML inference on the Neural Engine and GPU
- WebFormDriver for embedded WKWebView
- Profile building from PDF or text Label: value files
- Typed choice, yes/no and score decisions with probabilities
- Image decision models (Clef-vision, d3)
- Routing, safety and PII span detection (Vela 2.0)
- Multi-page document OCR to markdown
- Short reply drafting menu-bar demo
- CLI and SwiftUI demos

## Integrations
FluidAudio, Hugging Face, Core ML, macOS Accessibility API, WKWebView, vLLM Semantic Router, Cua

## Platforms
MACOS, IOS, WEB, API, VSC_EXTENSION, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
0.3.0

## Links
- Website: https://huggingface.co/FluidInference
- Repository: https://github.com/FluidInference/FluidUse
- EveryDev.ai: https://www.everydev.ai/tools/fluiduse