promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
Marker-Inc-Korea/AutoRAG
AutoRAG is a specialized tool that addresses the challenge of finding the most effective RAG pipeline for a given dataset and use case. It allows users to automatically evaluate various RAG modules and their combinations, providing a systematic way to identify the optimal pipeline without extensive manual effort. The framework supports the entire RAG optimization lifecycle, starting from data creation, which includes parsing and chunking documents, to generating QA datasets for evaluation. It then proceeds with RAG optimization, where it systematically explores different RAG pipeline configurations, applies various metrics to assess their performance, and helps users select the best-performing one. AutoRAG also offers functionalities to deploy the identified optimal RAG pipeline. This tool is particularly useful for ML practitioners and researchers who need to fine-tune RAG systems to achieve superior performance tailored to their specific data characteristics. The project's core functionality revolves around enhancing RAG system performance through automated evaluation and optimization, making it an essential tool for operating and improving AI agents that rely on RAG. It integrates with existing LLM and embedding models, and its modular design allows for flexibility in testing different components. AutoRAG directly contributes to the operational aspects of AI by providing a robust testing and optimization harness for a critical component of modern LLM applications.
https://github.com/Marker-Inc-Korea/AutoRAG
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.