promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
hegelai/prompttools
PromptTools, developed by Hegel AI, is an open-source, self-hostable library designed for the experimentation, testing, and evaluation of large language models (LLMs), vector databases, and prompts. It enables developers to compare prompts and parameters across various models, including OpenAI, Anthropic, LLaMA, and more, all within familiar interfaces such as code, Jupyter notebooks, or a local Streamlit-based playground. The tool supports evaluating the retrieval accuracy of vector databases like Chroma, Weaviate, Qdrant, LanceDB, and Pinecone. The core functionality revolves around creating and running experiments with different LLM providers, messages, models, and parameters, and then visualizing the results. PromptTools is unique in that it executes LLM calls directly from the user's machine, ensuring that no API keys, LLM inputs, or outputs are stored on remote servers or passed through intermediate services, addressing privacy and security concerns. It offers integrations with major LLM APIs and a growing list of vector databases, alongside support for frameworks such as LangChain, MindsDB, and LlamaIndex. The library also includes mechanisms for persisting experiment results to CSV, JSON, or MongoDB. Designed for developers, PromptTools focuses on providing a robust environment for prompt engineering and LLM application development, with active community support and a commitment to user data privacy.
https://github.com/hegelai/prompttools
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.