promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
rhesis-ai/rhesis
Rhesis is an open-source platform designed for collaborative testing of Large Language Models (LLMs) and agentic applications. It enables AI teams, including engineers, product managers, and domain experts, to work together on generating, simulating, and evaluating AI models. The platform offers AI-powered test generation, allowing users to describe requirements in natural language and synthesize hundreds of test scenarios, including edge cases and adversarial prompts, while also supporting knowledge-aware test generation by connecting to various context sources. Key features include single-turn and conversation simulation, with an agent named Penelope to simulate realistic conversations and test context retention, role adherence, and dialogue coherence. For adversarial testing, the Polyphemus Agent proactively identifies vulnerabilities like jailbreak attempts, prompt injection, PII leakage, and harmful content generation. It also integrates with Garak for comprehensive security testing. Rhesis provides over 60 pre-built evaluation metrics from frameworks like RAGAS and DeepEval, along with custom metrics, and offers LLM-as-Judge reasoning explanations. Observability features include OpenTelemetry-based tracing for monitoring LLM calls, latency, and token usage, linking traces directly to test results for debugging. The platform supports various LLM providers through LiteLLM integration, allowing users to bring their own models for test generation and evaluation. Rhesis caters to the entire testing lifecycle, from project setup and requirement definition to metric selection, test generation, execution, and team collaboration with integrated workflows and side-by-side comparisons. It aims to replace manual testing and traditional test frameworks by offering AI-generated test cases, handling non-deterministic outputs, and providing pre-production validation rather than just post-production monitoring.
https://github.com/rhesis-ai/rhesis
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.