promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
alphadl/AdaRubrics
AdaRubric addresses the limitations of static rubrics in evaluating goal-directed LLM agent tasks by providing a three-stage pipeline for dynamic, task-specific rubric generation and evaluation. It generates N orthogonal evaluation dimensions with calibrated 5-point scoring criteria based on task descriptions, caching rubrics for repeated use to reduce API costs. The tool scores each step of an agent's trajectory per-dimension with a confidence weight, using pluggable aggregators like Weighted Mean, Geometric Mean, or Min Score. A key innovation is its Data Filter stage, which includes a DimensionAwareFilter to curate high-quality DPO (Direct Preference Optimization) preference pairs. This prevents high-scoring dimensions from masking critical failures in other dimensions, a common issue with average scoring. AdaRubric significantly improves human correlation and inter-run reliability in agent evaluation, leading to notable gains in DPO task success and PPO convergence acceleration across various benchmarks. It is designed to be easily integrated into existing workflows, with straightforward installation and a clear pipeline for generating and applying dynamic rubrics for agent evaluation and reward learning.
https://github.com/alphadl/AdaRubrics
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.