Vision Agents

Vision Agents is an open-source Video AI framework designed for building real-time voice and video applications. It ships with Stream Video as its default low-latency transport, powered by a global edge network, while remaining edge/transport agnostic so developers can bring any edge layer they prefer. The framework makes it simple to prototype and scale a wide range of AI-powered video applications.

• Coaching & Training Applications — Build live sports coaching apps, guided workouts, and interactive training experiences with real-time video AI capabilities.

• Collaboration Tools — Create meeting assistants, automated note-taking systems, and transcription services for enhanced team productivity.

• Automation & Robotics — Develop IoT control systems, surveillance applications, and manufacturing workflow automation using video AI processing.

• Video AI Features — Build video avatars and character agents for interactive and engaging user experiences.

• 23+ Built-in AI Integrations — Connect with popular providers including OpenAI, Gemini, xAI, OpenRouter for LLMs; Deepgram, Fast-Whisper, Wizper for speech-to-text; ElevenLabs, Cartesia, AWS Polly for text-to-speech; and Ultralytics YOLO, Moondream, Roboflow for video processing.

• Realtime API Support — Leverage WebRTC connections through OpenAI, Gemini, AWS Bedrock, and Qwen for low-latency real-time interactions.

• Extensible Architecture — Build custom integrations using BaseProcessor or VideoProcessorMixin classes to plug in custom computer-vision models and extend functionality.

• Memory & Context Management — Utilize in-memory storage and Stream Chat integration for maintaining conversation context and state.

To get started, install Vision Agents and set up your first project following the installation guide. The framework provides comprehensive documentation covering voice agents, video agents, and integration setup. Developers can explore step-by-step implementation guides and ready-to-use cookbook examples for common use cases like building a golf coach application.