langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
VasiHemanth/tokentelemetry
TokenTelemetry offers a free, open-source, and entirely local observability dashboard designed for AI coding agents and autonomous agents like Hermes Agent. It provides a unified view to track critical metrics such as token usage, LLM API costs, and detailed session traces including prompts, reasoning chains, and tool calls. The tool supports various coding agents (e.g., Claude Code, Gemini CLI, OpenAI Codex CLI, GitHub Copilot) and features a dedicated dashboard for autonomous agents with insights into 38 source platforms, API call latency, cache hit rates, subagent summaries, and cost anomaly detection. Key features include real-time token usage dashboards, per-session and cumulative cost tracking, waterfall views of session traces, analytics on tool calls, per-project insights (heatmaps, activity timelines), model efficiency comparisons, and the ability to capture plan-mode outputs from agents. TokenTelemetry emphasizes privacy and ease of use, operating 100% locally with zero cloud dependency and automatic agent log detection. It helps developers understand agent behavior, optimize costs, and compare the efficiency of different models in their workflows, serving as a comprehensive monitoring solution for production AI systems.
https://github.com/VasiHemanth/tokentelemetry
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.