prompt2rig

prompt2rig is a universal-modder tool that turns a typed motion prompt into animation on a rigged character. A prompt goes to a video model on fal, a local GPU worker recovers 3D body and hand motion from the generated clip, and the result plays live on a Mixamo character in the browser. The project's own code is Apache-2.0 licensed.

What It Is

prompt2rig is a prompt-to-animation pipeline with a browser front end and a small Node server. You enter a prompt or choose a preset, and every clip starts from a generated performer image: one person, full body, plain backdrop, static camera. A video model (FLUX 3 Max or H3 Max Turbo on fal) acts out the motion. Motion is then lifted into 3D and retargeted onto a Mixamo rig. Takes can be exported as skinned, animated .glb files.

How the Motion Pipeline Works

A live pass runs MediaPipe Pose in the browser and retargets it per frame, so there is an immediate preview while the video is processed. The high-quality pass runs GVHMR on a local GPU worker to produce SMPL rotations mapped to Mixamo bones. WiLoR adds palm and 15 finger joints per hand, with a MediaPipe Hands fallback if the MANO model is not installed. The README lists timings measured on an RTX 4090 for a 5-second clip.

Using and Exporting

Built-in characters are X Bot, Michelle and Soldier, and any Mixamo .fbx or .glb in T-pose can be loaded. Stream mode chains clips into one continuous performance, with each clip starting from the last clip's final frame. Saved takes can be rendered headless in Blender into a studio-style video, and a capture tool records app sessions at 2560x1440. Prompts work best when the performer faces the camera with limbs clear of the body, such as dances, kicks and celebrations. Flips, spins, floor work and crossed arms are documented as hard for single-camera pose estimation.

Setup and Licensing Notes

Setup needs Node 18+ and a fal API key for the live preview. The HQ pass needs Windows, an NVIDIA GPU, uv and ffmpeg, plus SMPL-X and optionally MANO model downloads after free registration. The README states that the HQ components (GVHMR, SMPL-X, WiLoR, MANO) are research or non-commercial only, so commercial use would require replacing or licensing them. It was tested on Windows 11 with an RTX 4090.