promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
iMeanAI/WebCanvas
WebCanvas pioneers the online development, evaluation, and training of LLM-based web agents, addressing limitations of static, offline frameworks. It provides a comprehensive suite of toolkits for scaling and maintaining web trajectories, crucial for robust agent performance in dynamic web environments. The framework includes a base agent structure with plug-and-play modules for Planning, Observation, Memory, Reward, Action Execution, and Evaluation, allowing developers to easily test and iterate on their LLM-based web agents. Key features include dynamic and real-time web environment interaction, offering realistic assessment and feedback, and handling challenges like error handling and CAPTCHA solving. It introduces "key nodes" for granular, phase-based evaluation and provides tools to scale web agent evaluation in live web settings. WebCanvas also offers the Mind2Web-Live Dataset, a refined version of the Mind2Web static dataset, serving as a general-purpose benchmark for web agents. The project supports evaluation for various LLM services (OpenAI, Claude, Gemini) and provides open access to raw data for research, including HTML, screenshots, DOM trees, and operation videos. Recent updates include a JavaScript event listener-based evaluation system, integration with Browserbase for stable evaluation, and new metrics like USD consumption efficiency.
https://github.com/iMeanAI/WebCanvas
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.