promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
GiovanniPasq/chunky
Chunky is a local, open-source workspace designed to optimize document preparation for Retrieval-Augmented Generation (RAG) systems. It addresses common RAG pipeline failures originating from poorly processed source documents by providing a suite of tools for robust pre-processing. Key functionalities include PDF-to-Markdown conversion using multiple engines (PyMuPDF, Docling, MarkItDown, LiteParse, VLM, and Cloud API) to handle diverse document layouts and formats effectively, including scanned PDFs. The toolkit also provides capabilities for cleaning converted Markdown, inspecting document chunks visually, and comparing various chunking strategies (e.g., LangChain's token, recursive, character, and Markdown splitters, along with Chonkie's token, fast, sentence, recursive, table, code, semantic, and neural splitters, and Docling's hybrid and line-based strategies). This allows users to test and select the most appropriate chunking approach for their specific documents directly, instead of treating chunking as a hidden parameter. Furthermore, Chunky offers LLM-powered enrichment at both the Markdown and chunk levels, enabling the correction of conversion artifacts and the generation of context-aware titles, summaries, keywords, and retrieval questions to enhance document quality before indexing into a vector store. The platform supports batch processing, saving and reloading chunk versions, and a pluggable backend for adding custom converters or splitters, making it a comprehensive solution for improving the reliability and performance of RAG pipelines.
https://github.com/GiovanniPasq/chunky
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.