promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool designed to identify and address operational misalignment in AI models and agents before they impact business operations. It focuses on detecting issues that might not be captured by traditional KPIs, such as information leakage, fabricated citations, susceptibility to manipulative prompts, or unauthorized actions. The tool performs up to 45 inspections, categorized into 32 core graded inspections and 13 extended inspections for frontier risks like sabotage and oversight evasion. The core inspections evaluate five pillars of misalignment risk: fabrication, manipulation, deception, unpredictability, and opacity, producing a letter grade in under five minutes. iFixAi emphasizes trustworthiness, providing a repeatable diagnostic that can be integrated into CI/CD pipelines. It ensures that agents are judged independently, typically by external providers, rather than self-evaluation, though a self-judge mode is available for specific use cases. Each run generates a manifest of all inputs, allowing for exact replay of results. The tool supports various AI providers like OpenAI, Anthropic, OpenRouter, Gemini, Azure, Bedrock, and Hugging Face, enabling flexible evaluation across different models and platforms. It is designed to be industry and model agnostic, making it relevant for a wide range of AI deployments aiming for robust governance and safety. Users can also author their own fixtures to customize inspections.
https://github.com/ifixai-ai/iFixAi
Also in LLM Evaluation & Testing
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.
coze-dev/coze-loop
Cozeloop is an open-source platform offering full-lifecycle management for AI agents, encompassing development, debugging, evaluation, and monitoring.