langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
lmnr-ai/lmnr
Laminar is an open-source, Rust-powered observability platform specifically designed for AI agents and LLM-based applications. It provides a comprehensive suite of features to monitor, evaluate, and manage the performance and behavior of AI agents in production. Key functionalities include an OpenTelemetry-native tracing SDK that requires minimal code changes to instrument popular AI frameworks like Vercel AI SDK, LangChain, OpenAI, and Anthropic. This allows developers to gain deep insights into the execution flow of their agents. The platform also features an unopinionated and extensible SDK and CLI for running evaluations both locally and within CI/CD pipelines, complemented by a UI for visualizing and comparing evaluation results. For proactive issue detection, Laminar offers AI monitoring capabilities, enabling users to define events with natural language descriptions to track logical errors, custom behaviors, and overall health of their agents. All collected data, including traces, metrics, and events, is accessible via a built-in SQL editor and API, facilitating advanced analytics and dataset creation for further evaluations. Custom dashboards support SQL queries, providing flexible visualization options. Additionally, Laminar includes tools for data annotation and dataset creation, crucial for refining evaluation processes. Built with Rust, it emphasizes high performance, featuring a real-time engine for trace viewing, ultra-fast full-text search over span data, and a gRPC exporter for tracing. It supports both Python and TypeScript SDKs for easy integration.
https://github.com/lmnr-ai/lmnr
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.