langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK designed to provide a comprehensive platform for managing, optimizing, and safeguarding LLM projects and AI agents. It offers a wide array of features, including project management, dataset management, and evaluation management, allowing users to efficiently assess and protect their AI applications. The SDK supports robust trace management, enabling the recording and analysis of LLM application interactions, as well as specialized agentic tracing for monitoring and analyzing AI agent behavior. This includes tracking LLM interactions, token usage, tool utilization, network activities, user interactions, and agent decision-making processes. Key functionalities also encompass cost tracking, performance monitoring, and debugging for AI agents. Beyond observability and tracing, RagaAI Catalyst provides prompt management capabilities for efficient creation, storage, and retrieval of prompts, supporting versioning and variable handling. The platform also features synthetic data generation for testing, and guardrail management for establishing safety and compliance measures. Red-teaming functionalities are included to proactively identify vulnerabilities and biases in LLM applications. The SDK is designed for ease of integration with existing LLM workflows, offering both an SDK and a self-hosted dashboard for visualizing traces, debugging multi-agent systems, and performing advanced analytics with timeline and execution graph views. This makes it an end-to-end solution for MLOps needs specifically for the operational aspects of AI/ML models and agents.
https://github.com/raga-ai-hub/RagaAI-Catalyst
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.
latitude-dev/latitude-llm
Latitude is an open-source AI monitoring platform that provides issue detection, human-aligned evaluations, and agent-native tracing for LLM applications and AI agents.