promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
TIGER-AI-Lab/ClawBench
ClawBench is a comprehensive, open-source benchmark designed to evaluate the capability of AI browser agents in performing real-world, everyday online tasks. It features a diverse set of tasks (153 in V1, 130 in V2) across 144 live websites, covering approximately 15 life categories such as booking travel, ordering food, applying for jobs, and managing email. The benchmark measures end-to-end task success through a sophisticated 5-layer recording pipeline and utilizes an agentic evaluator that compares each run against human references. This approach provides a robust and realistic assessment of how well AI agents can navigate and interact with dynamic web environments to achieve specific goals. The project highlights that even the best agents currently complete only about one-third of the tasks, underscoring the challenges and areas for improvement in AI agent development. ClawBench provides a standardized framework for developers and researchers to test their AI agents, understand their limitations, and drive progress in the field of agentic AI. It also offers a sister project, HarnessBench, which focuses on fixing the base model and varying the harness, providing an orthogonal axis for evaluation. The project is designed for easy setup and execution, including a one-line quick start command and support for Docker-isolated harnesses. Overall, ClawBench serves as a crucial tool for both academic research and industry development in the rapidly evolving domain of AI agents, providing a quantitative measure of performance on practical, browser-based tasks.
https://github.com/TIGER-AI-Lab/ClawBench
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.