promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
superlinear-ai/raglite
RAGLite is a Python toolkit designed for building and operating Retrieval-Augmented Generation (RAG) systems. It emphasizes configurability, allowing users to select various components for their RAG pipeline, including Large Language Model (LLM) providers via LiteLLM (supporting local `llama-cpp-python` models), vector and keyword search databases like DuckDB or PostgreSQL (with `pgvector`), and rerankers. A core aspect of RAGLite is its focus on performance and advanced techniques, incorporating features such as multi-vector chunk embedding with late chunking and contextual chunk headings, optimal sentence splitting and semantic chunking using integer programming, and hybrid search methods that combine full-text search with vector similarity search. The toolkit also includes adaptive retrieval, where the LLM dynamically decides retrieval needs, prompt caching-aware message structures for cost and latency improvements, and adherence to Anthropic's long-context prompt format for enhanced output quality. It offers an optimal closed-form linear query adapter and is designed to be extensible with a built-in Model Context Protocol (MCP) server, an optional ChatGPT-like frontend via Chainlit, and document processing capabilities (PDF to Markdown, general document conversion with Pandoc, and OCR with Mistral OCR). Furthermore, RAGLite supports evaluation of retrieval and generation performance using Ragas, making it a comprehensive tool for developing, deploying, and monitoring RAG applications.
https://github.com/superlinear-ai/raglite
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.