langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
pydantic/logfire
Pydantic Logfire is an observability platform designed to provide deep insights into Python applications, particularly in production environments. It offers a dashboard for visualizing application behavior and is built as an opinionated wrapper around OpenTelemetry, supporting traces, metrics, and logs. This allows it to leverage existing OpenTelemetry tooling and integrations for various Python packages and even other languages. A key feature is its Python-centric approach, offering rich display of Python objects, event-loop telemetry, and profiling of Python code and database queries. It also features Pydantic Integration to understand data flow through Pydantic Validation models and gain built-in analytics on validations. Users can query their data using standard SQL, enabling integration with existing BI tools. While the SDKs (available for Python, TypeScript, and Rust) are open source and can export data to any OTel-compatible backend, the Logfire platform (UI and backend) is closed source, with an enterprise license option for self-hosting. This tool is particularly relevant for monitoring LLM and agent systems.
https://github.com/pydantic/logfire
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.