promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
agencyenterprise/PromptInject
PromptInject is a framework designed for the quantitative analysis of Large Language Models' (LLMs) robustness against adversarial prompt attacks. It provides a modular approach to assemble prompts, allowing researchers and developers to systematically test LLMs for vulnerabilities such as goal hijacking and prompt leaking. The framework enables the examination of how LLMs, like GPT-3, can be misaligned by simple, handcrafted inputs, highlighting long-tail risks. It's particularly useful for assessing an LLM's susceptibility to malicious user interactions that aim to derail its intended instructions or extract sensitive information. By offering a structured method to compose and execute these adversarial scenarios, PromptInject contributes to developing more secure and reliable LLM-powered applications. The project originated from the paper "Ignore Previous Prompt: Attack Techniques For Language Models," which received an award at NeurIPS ML Safety Workshop 2022, underscoring its academic rigor and practical relevance in the field of AI safety. This tool is essential for anyone involved in the deployment and operation of LLMs who needs to understand and mitigate potential adversarial exploits.
https://github.com/agencyenterprise/PromptInject
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.