promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
hermes-labs-ai/lintlang
LintLang is a static analysis tool for the configuration that defines how an AI agent behaves: MCP and function-tool definitions, parameter schemas, system prompts, messages, output contracts, and convention files such as AGENTS.md, CLAUDE.md, GEMINI.md, and SKILL.md, plus supported Python prompt code. Run locally with a single command, it inspects a project directory for this agent-facing content and flags defects that would otherwise only surface as unpredictable agent behavior at runtime: ambiguous tool choices where sibling tools overlap without a clear signal for the model to pick between them, conflicting requirements across instructions, gaps in declared schemas, and missing bounds on things like retries or loops. Because the analysis is deterministic and runs without invoking a model, it is meant to catch these setup defects before an agent is ever run against them, similar in spirit to a conventional code linter but targeted at the natural-language and schema surface that agents consume rather than source code syntax. The README illustrates this with a real finding from a merged pull request against an open-source agent project (ByteDance's DeerFlow), showing a flagged issue in a Vercel-related skill definition. The tool is distributed via PyPI, tracked by an OpenSSF Scorecard badge, and developed by Hermes Labs. It targets teams building or maintaining AI agents, MCP servers, or prompt-driven tooling who want an automated, repeatable check on the quality of an agent's instructions and tool interfaces, independent of the specific model that will execute them.
https://github.com/hermes-labs-ai/lintlang
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.