promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
Giskard-AI/giskard-oss
Giskard is an open-source Python library specifically designed for testing and evaluating AI agents and LLM applications. It provides a modular, lightweight, and async-first architecture. The library offers 'Giskard Checks' for creating and applying evaluations to test LLM-based systems, including scenario APIs, built-in checks, and LLM-as-judge capabilities. It helps catch regressions, validate RAG quality, enforce safety rules, and evaluate multi-turn agents by testing non-deterministic outputs. 'Giskard Scan' is the red-teaming layer that automatically generates adversarial test suites to detect vulnerabilities such as prompt injection, harmful content, and biases, based on OWASP LLM Top-10 threat categories. While Giskard v3 focuses on agentic systems, its v2 counterpart (still available) also includes features for automatic vulnerability detection ('Scan') and RAG evaluation test set generation ('RAGET') for both ML models and LLM applications. Giskard aims to provide comprehensive tools for ensuring the safety, performance, and reliability of AI systems in production.
https://github.com/Giskard-AI/giskard-oss
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
coze-dev/coze-loop
Cozeloop is an open-source platform offering full-lifecycle management for AI agents, encompassing development, debugging, evaluation, and monitoring.