promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
ShenSeanChen/waku-agent
Waku Agent is a meticulously designed, local-first personal AI assistant tool built with an emphasis on transparency and readability of its core components. It provides a clear, un-abstracted view into the four fundamental pillars of an AI agent: the Harness, the Loop, Memory management, and Evaluation/LLM-Ops. The project's primary goal is to serve as an educational blueprint, allowing users to understand and modify the underlying mechanisms of an AI assistant, rather than being a black-box product. Key features include a local-first design where all memory is stored in a single SQLite file, advanced memory management (semantic, episodic, and procedural memory with intelligent gating), a concise Python-based control loop (around 95 lines), and an integrated dashboard for real-time visualization of the agent's thought process. It also incorporates built-in evaluation capabilities, combining deterministic tests with LLM-as-judge mechanisms, and supports various LLM providers through a simple adapter. The dashboard provides insights into cost, latency, gate decisions, and allows interaction via multiple channels. The project differentiates itself from commercial products by prioritizing ownership and understanding of the codebase over end-user convenience, providing an accessible reference for agent architectural patterns.
https://github.com/ShenSeanChen/waku-agent
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.