# framedex

> Open-source Claude Code skill that turns video and photo archives into a queryable, plain-text knowledge base of per-file description sidecars.

framedex is an open-source project from SimbaStack that indexes video clips and still photos across folders, SSDs, or an Apple Photos library. For each file it writes a plain-text `.description.md` sidecar next to the original, which is never modified. It installs the `fdx` command-line tool and runs as a Claude Code skill. The latest listed release is v0.2.0.

## What It Is

framedex is a local-first, resumable indexing pipeline that converts a scattered media archive into a portable knowledge base. Video clips get GPS location and reverse-geocoded place name, a speaker-diarized multilingual transcript, an English translation when needed, face detection, and an AI vision scene description with a keep/review/cull rating. Still photos get the same treatment minus audio, plus a camera, lens, and exposure block read from EXIF. RAW formats are read from their embedded JPEG previews.

## How the Pipeline Works

Per clip, it uses ffprobe and exiftool for metadata and GPS, Nominatim for place names, ffmpeg to sample five content-diverse frames and extract audio, WhisperX for transcription, alignment and pyannote diarization, insightface for face embeddings, and a vision model for a structured Scene/Subjects/Action/Mood/Shot type/Use cases description. For photos, bursts and RAW+JPEG pairs share one vision assessment per group, with stub sidecars for alternates. Writes are atomic and already-indexed files are skipped on re-runs.

## Querying and Export

Companion commands cover the rest of the workflow:

- `fdx-query` filters sidecars by rating, keyword, person, location, language, and more.
- `fdx-master` produces a drive-level `_INDEX.md` and `_INDEX.json`.
- `fdx-summary` writes per-folder summaries.
- `fdx-xmp` exports ratings, keywords, and captions to `.xmp` sidecars for Lightroom with proprietary RAW files.
- `fdx-photos` indexes an Apple Photos library on macOS.
- `fdx-mcp` serves an indexed archive over stdio to MCP hosts such as Claude Desktop or LM Studio, with six tools including query, sidecar reading, contact sheets, and user ratings.

## Vision Backends and Privacy

The vision step can use the `claude -p` CLI, the Anthropic API, or a local OpenAI-compatible server such as LM Studio. The cli and api backends send frames to Anthropic, while the local backend stays fully on-device. Local components include ffmpeg, exiftool, Whisper, pyannote, and insightface, and the face database stays local. Nominatim geocoding sends only coordinates and can be skipped.

## Known Limitations

The README notes that diarization degrades on heavy ambient noise, WhisperX runs on CPU on Apple Silicon, face cluster IDs are temporary hashes until a labeling tool ships, and `fdx-photos` does not yet group bursts.

## Features
- Per-file .description.md sidecars; originals never modified
- GPS extraction with reverse-geocoded place names
- Speaker-diarized multilingual transcripts with English translation
- Face detection and embeddings
- AI scene description with keep/review/cull rating
- Photo support including RAW, JPEG, HEIC with EXIF camera block
- Burst and RAW+JPEG pair grouping
- Resumable, atomic, idempotent indexing
- fdx-query filtering by rating, keyword, person, place, language
- Drive-level master index (_INDEX.md and _INDEX.json)
- Lightroom .xmp sidecar export for ratings and keywords
- Apple Photos library indexing via fdx-photos
- MCP server (fdx-mcp) with six tools
- Cloud or fully local vision backends

## Integrations
Claude Code, Anthropic API, LM Studio, Claude Desktop, MCP, Apple Photos, Adobe Lightroom Classic, Hugging Face pyannote, WhisperX, ffmpeg, exiftool, Nominatim

## Platforms
MACOS, LINUX, API, CLI

## Pricing
Open Source

## Version
v0.2.0

## Links
- Website: https://github.com/Simbastack-hq/framedex
- Documentation: https://github.com/Simbastack-hq/framedex/tree/main/docs
- Repository: https://github.com/Simbastack-hq/framedex
- EveryDev.ai: https://www.everydev.ai/tools/framedex