promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
PacificAI/langtest
LangTest is an open-source library designed for comprehensive testing and evaluation of Large Language Models (LLMs) and other Natural Language Processing (NLP) models. It allows users to generate and execute over 60 distinct types of tests with minimal code, focusing on critical quality aspects such as robustness, bias, representation, fairness, and accuracy. The library supports popular NLP frameworks like Spark NLP, Hugging Face, and Transformers, and integrates with various LLM providers including OpenAI, Cohere, AI21, and Azure-OpenAI. Key functionalities include testing LLMs for question answering, toxicity, clinical relevance, legal support, factuality, sycophancy, and summarization. LangTest also provides capabilities to automatically augment training data based on test results for select models, aiding in the continuous improvement of model performance and safety. It comes equipped with a variety of benchmark datasets tailored for different use cases and evaluation scenarios, ensuring models are challenged and refined across diverse linguistic tasks. The project emphasizes Responsible AI, offering tools to detect and mitigate issues like demographic bias and gender-occupational stereotypes in AI outcomes.
https://github.com/PacificAI/langtest
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.