promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
plurai-ai/intellagent
IntellAgent is an advanced multi-agent framework designed to evaluate and optimize conversational AI agents comprehensively. It achieves this by simulating realistic, challenging interactions, generating thousands of edge-case scenarios tailored to the specific agent under test. The framework allows for the simulation of diverse user interactions across varying complexity levels, providing a robust testing environment. After simulation, IntellAgent delivers comprehensive performance evaluations, offering detailed analysis to identify performance gaps, prioritize improvements, and compare outcomes across different experiments. The system operates in three main steps: first, it decomposes a user prompt into a policy graph, samples relevant policies based on real conversation distributions, and generates a user-chatbot interaction scenario. Second, it simulates this interaction using a dedicated user agent. Finally, it critiques the conversation and provides feedback on the tested policies. Key features include automatic generation of edge-case scenarios, simulation of diverse user interactions, and comprehensive performance evaluations. IntellAgent aims to enhance agent performance, reliability, and user experience, ensuring agents are ready for reliable real-world deployment. It emphasizes simple integration with existing conversational agents and includes tools for visualizing simulation results through a Streamlit dashboard, offering detailed analytics.
https://github.com/plurai-ai/intellagent
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.