promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
truera/trulens
TruLens is an open-source framework designed for the evaluation and tracking of Large Language Model (LLM) applications and AI agents. It provides tools for systematically understanding and improving the performance of LLM-powered applications, covering aspects from prototype development to iteration and comparison of different versions. The platform offers fine-grained, stack-agnostic instrumentation built on OpenTelemetry, capturing every function call, LLM generation, retrieval, and tool invocation as structured OTEL spans. This allows for interoperability with existing observability infrastructure, enabling export of traces to various OTLP-compatible backends like Jaeger, Grafana Tempo, and Datadog. Key features include purpose-built evaluators for agentic systems, measuring aspects like logical consistency, execution efficiency, plan adherence, plan quality, tool selection, tool calling, and tool quality. TruLens supports both inline evaluation, where feedback runs alongside the application, and batch evaluation for pre-collected datasets. It includes Model Context Protocol (MCP) support for instrumenting tool calls, capturing details like tool name, arguments, output, and latency. A flexible Selector API allows targeting specific span attributes for evaluation. TruLens integrates with various LLM providers (e.g., OpenAI, LiteLLM, Google Gemini, AWS Bedrock, HuggingFace) and application frameworks like LangChain and LlamaIndex, making it a versatile tool for LLMOps and MLOps workflows focused on the operational aspects of AI.
https://github.com/truera/trulens
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.