langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
latitude-dev/latitude-llm
Latitude is an open-source AI monitoring platform designed to help developers and operators understand, debug, and improve the performance of their LLM applications and AI agents. It functions similarly to Sentry, but is purpose-built for the intricacies of AI/ML models in production. Key features include issue-centric monitoring, which groups failed traces into trackable issues with status and trend analysis. It offers human-aligned evaluations, automatically building eval metrics from team judgments and tracking drift from human alignment over time. The platform provides agent-native traces, allowing users to visualize multi-turn sessions, tool calls, and full execution paths in a single view. Latitude also integrates semantic search, enabling users to find any trace by meaning or exact matches across 100% of their data, without sampling. It supports issue alerts through Slack, email, or webhooks for immediate notification of new or escalating issues. Additionally, an MCP (multi-cloud platform) server allows direct interaction with Latitude's capabilities from coding agents. The platform is provider-agnostic, supporting various LLM providers and frameworks like OpenAI, Anthropic, Bedrock, and LangChain, and is compatible with any OpenTelemetry (OTLP) backend. It can be self-hosted on Docker Compose or Kubernetes via Helm charts.
https://github.com/latitude-dev/latitude-llm
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.