langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
cyberark/agentwatch
Agentwatch is an advanced observability framework designed to provide deep insights into the runtime behavior of AI agents. It focuses on tracking and analyzing interactions within AI-driven applications, particularly those involving Large Language Models (LLMs) and various tools. The framework offers comprehensive interaction tracking, monitoring LLM calls and tool usage in real-time. It captures detailed metadata, including tool inputs, arguments, and performance metrics, to provide a complete picture of agent operations. Agentwatch supports multiple AI development frameworks, including Langgraph and Autogen, and is designed to be compatible with any Python-based application utilizing LLMs. A key feature is its one-liner observability integration, allowing developers to gain instant insights with minimal setup. It also provides advanced visualization capabilities to generate intuitive graphs for in-depth analysis of agent behavior. The project's primary purpose is to help developers optimize their AI-driven applications by offering unprecedented visibility and understanding of how agents interact with their environment and perform tasks.
https://github.com/cyberark/agentwatch
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.