promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
huangyiminghappy/ai-eval-platform
This project provides a comprehensive, self-hosted AI evaluation platform designed for an engineering workflow, moving beyond one-off metric scripts. It integrates various evaluation aspects into a single Web UI, including dataset management, scenario presets, native metrics, OpenAI-compatible judge models, live endpoint evaluation, detailed report analysis, manual review, and A/B blind testing. The platform supports evaluating diverse LLM applications such as RAG systems, AI Agents, and multi-turn conversational models. Key features include a web UI for managing datasets, scenarios, metrics, execution, reports, and human evaluations; support for both offline and live endpoint evaluations; built-in metrics specifically tailored for RAG (e.g., faithfulness, context recall), Agents (e.g., task completion, tool correctness), and multi-turn conversations (e.g., topic adherence, knowledge retention); and robust human blind testing and manual review functionalities to inform release decisions. It emphasizes repeatable evaluation operations, allowing users to collect samples, freeze scenario snapshots, run comparable tasks, inspect row-level reasons, and combine automatic judging with human decisions. The platform is designed to be lightweight with a SQLite quick start and supports PostgreSQL for larger datasets, making it suitable for teams looking to standardize their LLM application evaluation processes.
https://github.com/huangyiminghappy/ai-eval-platform
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.