promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
langwatch/better-agents
Better Agents is a command-line interface (CLI) tool and a set of established standards designed to streamline the development and deployment of AI agents. It aims to supercharge coding assistants like Kilocode, Claude Code, and Cursor by enabling them to act as experts in various agent frameworks (e.g., Agno, Mastra, LangGraph) and adhere to best practices. The project provides a standardized structure for agent projects, including dedicated directories for agent code, tests (scenario tests and evaluation notebooks), and versioned prompts. Key features include end-to-end scenario testing to ensure agent behavior, prompt versioning for collaboration, evaluation notebooks for measuring prompt performance, built-in observability instrumentation (via LangWatch), and a structured approach to maintainability. The CLI guides users through project initialization, helping them select programming languages, agent frameworks, coding assistants, LLM providers, and API keys. It emphasizes an "Agent Testing Pyramid" philosophy for robust development. Better Agents connects with LangWatch for observability and prompt management, and provides tools for offline evaluation and testing, ensuring that agents are well-tested and ready for production environments. It requires Node.js 22+, npm/pnpm, a compatible coding assistant, LangWatch API key, and an LLM provider API key.
https://github.com/langwatch/better-agents
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.