# MulmoChat

> A research prototype exploring multimodal AI chat experiences where users converse naturally while experiencing rich visual and interactive content on a canvas.

MulmoChat is a research prototype that reimagines AI chat interfaces by combining natural language conversation with rich visual and interactive content rendered directly on a canvas. Unlike traditional text-based chat, MulmoChat enables images to materialize, maps to become explorable, and games to turn playable—all within the conversational flow. Built with TypeScript and Vue, it supports multiple LLM providers and local image generation via ComfyUI integration.

- **Multimodal Canvas Interface** — *Users engage in voice or text chat while visual content (images, maps, interactive elements) renders simultaneously on a canvas alongside the conversation.*
- **Voice Chat Support** — *Click "Start Voice Chat" to interact with the AI using your microphone; the system processes speech and responds with both text and visual output.*
- **Provider-Agnostic Text API** — *A unified `/api/text/generate` endpoint supports OpenAI, Anthropic, Google Gemini, and Ollama, normalizing responses across vendors.*
- **ComfyUI Local Image Generation** — *Integrates with ComfyUI Desktop for local image generation using FLUX and Stable Diffusion models, with automatic model-specific parameter optimization.*
- **Plugin Architecture** — *Developers can extend MulmoChat with new capabilities by implementing TypeScript contracts, Vue views, and configuration as described in TOOLPLUGIN.md.*
- **Multi-Provider LLM Support** — *Configure API keys for OpenAI, Anthropic, Gemini, and Ollama; each provider can be enabled independently via environment variables.*
- **AI-Powered Search** — *Optional Exa integration enables AI-powered web search within the conversational interface.*
- **Google Maps Integration** — *Optional Google Maps API key enables explorable map experiences directly within the chat canvas.*
- **Getting Started** — *Run `yarn install`, create a `.env` file with your API keys, then `yarn dev` to launch the development server and open the browser.*

## Features
- Multimodal canvas chat interface
- Voice chat support
- Provider-agnostic text generation API
- ComfyUI local image generation
- Plugin/tool extension architecture
- OpenAI, Anthropic, Gemini, and Ollama support
- AI-powered search via Exa
- Google Maps integration
- FLUX and Stable Diffusion model support
- Unified text API with normalized responses

## Integrations
OpenAI, Anthropic, Google Gemini, Ollama, ComfyUI, Exa, Google Maps

## Platforms
WEB, API, CLI

## Pricing
Open Source

## Version
0.1.1

## Links
- Website: https://github.com/receptron/MulmoChat
- Documentation: https://github.com/receptron/MulmoChat
- Repository: https://github.com/receptron/MulmoChat
- EveryDev.ai: https://www.everydev.ai/tools/mulmochat