promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
LeoYeAI/myclaw-bench
MyClaw Bench is a dedicated benchmarking suite designed to rigorously evaluate the performance of AI agents, particularly those operating within the OpenClaw environment. Unlike many benchmarks that focus on simple format compliance, MyClaw Bench emphasizes real-world task outcomes, complex reasoning, safety, efficiency, resilience, and consistency. It comprises 45 tasks distributed across four difficulty tiers: Foundation (basic capabilities), Reasoning (handling ambiguity and multi-step chains), Mastery (trustworthiness and error recovery), and the highly discriminating Frontier and Computer Use tiers (advanced reasoning, metacognition, and interactive browser use). The benchmark provides a composite score based on success rate, efficiency, safety, consistency, and a distinct weighting for Frontier tasks. It leverages semantic grading over simple regex matching, utilizes fixed time contexts, and supports running tests with various models, specific tiers, or individual tasks. The project provides a script for easy execution, configurable parameters for runs and judge models, and requires a Python 3.10+ environment with uv package manager, a running OpenClaw instance, and model API keys. It adheres to design principles that prioritize semantic correctness, resilience to tool failures, and the ability to handle under-specified tasks, making it a robust tool for assessing and comparing AI agent capabilities in production-like scenarios.
https://github.com/LeoYeAI/myclaw-bench
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.