promptfoo/promptfoo
Promptfoo is a CLI and library for evaluating LLM applications, offering automated testing, red teaming, and vulnerability scanning for prompts, models, agents, and RAGs.
Awesome Infra for AI › LLM Evaluation & Testing
juanjuandog/FinSight-AI
FinSight AI provides a robust, backend-heavy framework for building dependable AI research systems, moving beyond simple RAG demonstrations. It is purpose-built for serving and operating AI agents, focusing on the inference and runtime aspects rather than model training. The project features long-running agent workflows with explicit state transitions, idempotent task submission, and sophisticated concurrency control using Redis Lua single-flight leases. It implements report caching tied to data snapshots, ensuring trustworthiness and reusability of AI-generated reports. The system utilizes PostgreSQL and pgvector for hybrid retrieval with evidence traceability, which is crucial for grounded answers, report citations, and hallucination risk checks. Furthermore, FinSight AI includes comprehensive RAG and agent quality evaluation for regression checks, assessing hit rate, evidence coverage, answer coverage, and hallucination risk. The architecture integrates Spring Boot as the backend, RabbitMQ for asynchronous workflows, and a FastAPI AI service with Ollama fallback, demonstrating a full production-ready setup for AI agent operations. It's designed to show how to build the infrastructure around an AI agent, offering features like a runnable institutional research console that exposes the research workflow, report cache, evidence trace, RAG evaluation, and financial signals produced by the backend. This project aligns with the operational side of AI by providing the tooling and infrastructure necessary to deploy, run, and evaluate AI agents in a production environment.
https://github.com/juanjuandog/FinSight-AI
Also in LLM Evaluation & Testing
ifixai-ai/iFixAi
iFixAi is a diagnostic tool that evaluates AI models and agents for operational misalignment, including fabrication, manipulation, deception, unpredictability, and opacity, by running up to 45 insp...
confident-ai/deepeval
DeepEval is an open-source LLM evaluation framework, offering a variety of metrics and tools for assessing the performance of AI agents, RAG pipelines, and chatbots through unit testing.
vibrantlabsai/ragas
Ragas is an evaluation framework for LLM applications that provides objective metrics, test data generation, and feedback loops for continuous improvement.
NVIDIA/garak
Garak is an open-source LLM vulnerability scanner designed to red-team and assess generative AI models for weaknesses like hallucination, data leakage, prompt injection, and toxicity.
evidentlyai/evidently
Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM systems, providing comprehensive data and model quality checks from experiments to production.
open-compass/opencompass
Open-source LLM evaluation platform for running standardized benchmarks across models, with configurable datasets, prompt templates, and an official public leaderboard.
Giskard-AI/giskard-oss
Giskard is an open-source Python library for testing and evaluating agentic systems and LLM applications, offering tools for scenario-based testing, red teaming, and vulnerability scanning.