promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
coze-dev/coze-loop
Cozeloop is an open-source, developer-centric platform designed to streamline the development and operation of AI agents. It provides comprehensive lifecycle management capabilities, addressing challenges from the initial development phase through debugging, evaluation, and continuous monitoring in production. Key features include a visual Playground for prompt development, allowing real-time interactive testing, debugging, and version management of prompts with support for comparing outputs from different LLMs. The platform also offers a robust evaluation module, enabling automated and multi-dimensional testing of prompts and AI agent outputs for metrics such as accuracy, conciseness, and compliance. Furthermore, Cozeloop provides deep observability into the entire execution process of AI agents, recording every stage from user input to AI output, including prompt parsing, model invocation, tool execution, and automatic capture of intermediate results and exceptions. This detailed tracing helps developers understand agent behavior and diagnose issues effectively. Cozeloop supports integration with various models like OpenAI and Volcengine Ark and provides deployment options via Docker Compose and Kubernetes Helm charts, making it accessible for diverse development and operational environments. By offering its core technological framework as open source, Cozeloop encourages community contributions and extensions.
https://github.com/coze-dev/coze-loop
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.