langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
comet-ml/opik
Opik, developed by Comet, provides a full lifecycle management solution for generative AI applications, from prototyping to production. It offers extensive tracing capabilities for LLM calls, conversation logging, and agent activity, with broad integration support for popular frameworks like Google ADK, Autogen, and Flowise AI. The platform includes advanced evaluation features such as automated prompt evaluation, LLM-as-a-judge metrics for tasks like hallucination detection and moderation, and functionalities for RAG assessment (Answer Relevance, Context Precision). Developers can manage datasets and experiments, and integrate evaluations into CI/CD pipelines using its PyTest integration. For production environments, Opik supports scalable monitoring dashboards, capable of logging over 40 million traces per day, and enables the use of online evaluation rules. Additionally, it features the Opik Agent Optimizer for enhancing prompts and agents, and Opik Guardrails to implement safe and responsible AI practices, ensuring robust and optimized LLM deployments.
https://github.com/comet-ml/opik
Also in LLM Observability & Tracing
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.
latitude-dev/latitude-llm
Latitude is an open-source AI monitoring platform that provides issue detection, human-aligned evaluations, and agent-native tracing for LLM applications and AI agents.