framedex

framedex is an open-source project from SimbaStack that indexes video clips and still photos across folders, SSDs, or an Apple Photos library. For each file it writes a plain-text .description.md sidecar next to the original, which is never modified. It installs the fdx command-line tool and runs as a Claude Code skill. The latest listed release is v0.2.0.

What It Is

framedex is a local-first, resumable indexing pipeline that converts a scattered media archive into a portable knowledge base. Video clips get GPS location and reverse-geocoded place name, a speaker-diarized multilingual transcript, an English translation when needed, face detection, and an AI vision scene description with a keep/review/cull rating. Still photos get the same treatment minus audio, plus a camera, lens, and exposure block read from EXIF. RAW formats are read from their embedded JPEG previews.

How the Pipeline Works

Per clip, it uses ffprobe and exiftool for metadata and GPS, Nominatim for place names, ffmpeg to sample five content-diverse frames and extract audio, WhisperX for transcription, alignment and pyannote diarization, insightface for face embeddings, and a vision model for a structured Scene/Subjects/Action/Mood/Shot type/Use cases description. For photos, bursts and RAW+JPEG pairs share one vision assessment per group, with stub sidecars for alternates. Writes are atomic and already-indexed files are skipped on re-runs.

Querying and Export

Companion commands cover the rest of the workflow:

• fdx-query filters sidecars by rating, keyword, person, location, language, and more.
• fdx-master produces a drive-level _INDEX.md and _INDEX.json.
• fdx-summary writes per-folder summaries.
• fdx-xmp exports ratings, keywords, and captions to .xmp sidecars for Lightroom with proprietary RAW files.
• fdx-photos indexes an Apple Photos library on macOS.
• fdx-mcp serves an indexed archive over stdio to MCP hosts such as Claude Desktop or LM Studio, with six tools including query, sidecar reading, contact sheets, and user ratings.

Vision Backends and Privacy

The vision step can use the claude -p CLI, the Anthropic API, or a local OpenAI-compatible server such as LM Studio. The cli and api backends send frames to Anthropic, while the local backend stays fully on-device. Local components include ffmpeg, exiftool, Whisper, pyannote, and insightface, and the face database stays local. Nominatim geocoding sends only coordinates and can be skipped.

Known Limitations

The README notes that diarization degrades on heavy ambient noise, WhisperX runs on CPU on Apple Silicon, face cluster IDs are temporary hashes until a labeling tool ships, and fdx-photos does not yet group bursts.