promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
vectara/open-rag-eval
Open RAG Eval is an open-source Python toolkit designed for evaluating and improving Retrieval-Augmented Generation (RAG) pipelines. It provides a flexible and extensible framework to measure the performance of RAG systems, helping users identify areas for improvement. A key feature is its ability to perform RAG evaluation without the need for "golden answers" or golden chunks, leveraging techniques like UMBRELA and AutoNuggetizer. This makes RAG evaluation more scalable and accessible by removing a common bottleneck. The toolkit also supports optional golden answer evaluation with appropriate metrics when reference answers are available. Out-of-the-box, it includes implementations of TREC-RAG evaluation metrics and connectors for popular RAG platforms and frameworks such as Vectara, LlamaIndex, and LangChain. Its modular architecture allows for easy integration of custom evaluation metrics and pipelines. The tool generates detailed per-query scores and intermediate outputs for debugging and analysis, and also offers plotting utilities for visualizing and comparing results across different configurations. Prerequisites include Python 3.9+, an OpenAI API key, and optionally a Hugging Face token for the open-source HHEM model or a Vectara API key for its commercial factual consistency API. Installation is straightforward via pip, supporting both command-line usage and development from source.
https://github.com/vectara/open-rag-eval
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.