promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
athina-ai/athina-evals
Athina-evals is an open-source Python SDK designed for evaluating the quality and performance of Large Language Model (LLM) generated responses. It provides a comprehensive suite of over 50 preset evaluation metrics and also supports the creation of custom evaluation logic, allowing users to tailor assessments to their specific needs. This SDK is a core component of the Athina Observability and Experimentation platform for AI teams. It facilitates programmatic execution of LLM evaluations, enabling developers to integrate assessments directly into their development workflows. Beyond programmatic use, the SDK seamlessly integrates with the Athina IDE, a web-based environment where users can visualize evaluation results, prototype LLM pipelines, run various experiments, and perform side-by-side comparisons of different datasets. This integration helps in identifying performance regressions, understanding model behavior, and making data-driven decisions during the development and deployment of LLM applications. The tool focuses specifically on the post-inference evaluation of LLMs, making it a critical asset for ensuring the reliability and effectiveness of AI agents and LLM-powered systems in production environments.
https://github.com/athina-ai/athina-evals
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.