langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
stainlu/hermes-labyrinth
Hermes Labyrinth is a specialized observability plugin designed for the Hermes Agent, offering a read-only view into the agent's autonomous operations. It transforms complex agent activities into a visual 'Labyrinth map' that chronicles 'crossings' such as prompts, tool calls, tool results, failures, model switches, subagent interactions, memory accesses, and context compressions. Key features include a 'Journey index' for tracking recent work, an 'Inspector' for detailed crossing analysis, 'Guideposts' for observations backed by local evidence, and a 'Skill atlas' for understanding effective and shadowed skills. It also provides insights into cron-triggered autonomy, model transitions via a 'Model ferry,' and generates redacted Markdown and JSON reports for individual journeys. The project focuses on transparently recording and visualizing agent behavior, emphasizing data policy with read-only access and robust secret redaction capabilities to ensure sensitive information is not exposed in trace data or reports. It's a critical tool for understanding and debugging the internal workings of autonomous AI agents in production.
https://github.com/stainlu/hermes-labyrinth
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.