langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
Javis603/token-monitor
Token Monitor is a desktop widget designed to provide real-time visibility into the operational aspects of AI coding tools. It tracks live token usage, monitors AI service limits, and calculates costs for a range of integrated AI assistants, including Claude Code, Codex, Hermes Agent, OpenCode, OpenClaw, Cursor, Antigravity, and GitHub Copilot CLI among others. A key feature is its multi-device synchronization, allowing users to track usage across different machines via a self-hosted sync backend or a Cloudflare Worker. The widget offers detailed breakdown views, allowing users to analyze usage by tool, device, model, or session. It also provides per-session details, showing token usage per prompt and replies, cache hit statistics, and cost breakdowns alongside token counts. For operational oversight, it includes AI tool limits detection for various services, covering session, weekly, billing windows, and prepaid balances. An optional status view provides information on service availability for major AI providers. The tool emphasizes privacy, only syncing summary numbers, and supports a local-first mode for single-device use. Its primary purpose revolves around observing and managing the consumption and operational status of AI models and agents during active use.
https://github.com/Javis603/token-monitor
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.