promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
JudgmentLabs/judgeval
Judgeval provides a comprehensive suite of tools for the continuous improvement and operational management of AI agents. It leverages OpenTelemetry for robust tracing, allowing developers to instrument functions and automatically capture inputs, outputs, and LLM token usage, ensuring compatibility with existing observability stacks. A core feature is its agent-judge system, which defines prompt-based scorers to evaluate agent behaviors at scale. These judges produce structured outputs that describe agent actions, accumulating into searchable records that facilitate understanding of system performance. Judgeval supports online monitoring by automatically scoring live production traffic without latency impact, surfacing detected behaviors as structured signals and enabling configurable alerts for regressions. The SDK offers broad integrations with major LLM providers like OpenAI, Anthropic, Google GenAI, and Together AI, along with framework support for LangGraph, OpenLit, and Claude Agent SDK. It also includes a CLI for managing agents, traces, judges, and evaluations from the terminal, enabling querying of trace history, deployment of judges, and inspection of behaviors. Additionally, the MCP (MLOps Connectivity Protocol) Server connects Judgeval to any MCP-compatible AI tool, allowing users to query agent traces, invoke judges, browse behaviors, and identify failures directly within AI assistants or IDEs. This makes Judgeval an end-to-end solution for observing, evaluating, and refining the performance of AI agents in production.
https://github.com/JudgmentLabs/judgeval
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.