Agenta

Agenta is an open-source LLMOps platform that helps developers and product teams build reliable LLM applications. It covers the LLM development lifecycle with tools for prompt management, evaluation, and observability. Agenta provides a web UI, APIs, and SDKs so teams can collaborate, run systematic evaluations, and monitor production behavior.

• Prompt management: Organize, version, and collaborate on prompts so subject-matter experts can edit prompts outside the codebase; use the playground to iterate and run side-by-side comparisons.
• Evaluation: Run automatic evaluations at scale, human annotation workflows, and online evaluation against production traffic to validate changes before and after deployment.
• Observability & tracing: Capture traces and user feedback via API, debug agent execution flows, and track cost and failure cases over time to add to test sets.
• Playground and test sets: Experiment with prompts and test sets inside the UI to reproduce and fix edge cases, then promote validated prompts to production.
• Integrations and SDKs: Integrate with major LLM providers and frameworks via API and SDKs; Agenta is OpenTelemetry-compliant for tracing and supports adding custom or self-hosted models.

To get started, sign up for the web product or self-host the MIT-licensed project, add your model providers and prompts, create test sets, and run evaluations from the UI or via the API/SDK.