PromptTools

PromptTools is an open-source toolkit from Hegel AI for experimenting with and evaluating prompts, LLMs, and vector databases using code, notebooks, and a local playground. The latest documented release is v0.0.45, published in January 2024, which the release name describes as introducing observability features.

What It Is

PromptTools is a Python library for developers who want to test prompts and model parameters through familiar interfaces. In a few lines of code, you define an experiment with models, messages, and parameters such as temperature, run it, and visualize the outputs side by side. The same approach covers evaluating retrieval accuracy of vector databases.

Workflow and Outputs

Experiments can be run in Jupyter notebooks, including in Google Colab, or through a Streamlit-based playground launched locally. A hosted playground is also available on Streamlit Community Cloud, though it does not support LlamaCpp. Evaluations can be turned into unit tests and integrated into CI/CD via GitHub Actions. Results can be persisted by exporting an experiment to CSV, JSON, LoRA JSON, or MongoDB.

Supported Integrations

Documented LLM support includes OpenAI, LLaMA.Cpp, HuggingFace, Anthropic, Mistral AI, Google Gemini, Vertex AI, Azure OpenAI Service, and Replicate. Vector database support includes Chroma, Weaviate, Qdrant, LanceDB, and Pinecone. It also supports LangChain and MindsDB frameworks, plus Stable Diffusion for computer vision.

Privacy and Data Handling

The documentation states that code runs on the user's machine and LLM API calls are made directly from it, with no forwarding. It also states API keys and LLM inputs and outputs stay local. Anonymous error usage data is sent to Sentry, and users can opt out with the SENTRY_OPT_OUT environment variable.