langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
traceloop/openllmetry
OpenLLMetry is an open-source project that extends OpenTelemetry to provide comprehensive observability for Generative AI and LLM-powered applications. It enables developers to gain insights into the runtime behavior of their AI systems by instrumenting interactions with LLM providers (e.g., OpenAI, Anthropic, Google Generative AI), vector databases (e.g., Chroma, Pinecone, Qdrant), and other AI frameworks. The project leverages existing OpenTelemetry standards to ensure compatibility with a wide range of observability backends, including Datadog, Honeycomb, Grafana, and Splunk, among others. By integrating OpenLLMetry, users can collect and export detailed traces and metrics, allowing for effective monitoring, debugging, and performance optimization of their LLM applications in production. The Traceloop SDK simplifies the setup process, enabling quick instrumentation with just a few lines of code. This tool is crucial for understanding the operational nuances of conversational AI agents, RAG systems, and other generative AI solutions, helping identify issues related to latency, cost, and API usage across various components of an LLM stack.
https://github.com/traceloop/openllmetry
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.
latitude-dev/latitude-llm
Latitude is an open-source AI monitoring platform that provides issue detection, human-aligned evaluations, and agent-native tracing for LLM applications and AI agents.