langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
Agenta-AI/agenta
Agenta is a comprehensive open-source LLMOps platform that streamlines the process of building and deploying production-grade LLM applications. It provides a suite of tools for prompt management, including an interactive LLM playground for side-by-side prompt comparison, multi-model support (50+ models and custom model integration), version control for prompts and configurations with branching and environments, and support for complex configuration schemas. The platform also offers robust LLM evaluation capabilities, allowing users to systematically assess their LLM applications with both human and automated feedback. This includes flexible test set creation from production data or playground experiments, pre-built and custom evaluators (including LLM-as-judge), UI and API access for running evaluations, and integrated human feedback collection. Furthermore, Agenta provides LLM observability features to deliver insights into LLM applications in production, such as cost and performance tracking, detailed LLM tracing for debugging complex workflows, and support for open standards like OpenTelemetry, OpenLLMetry, and OpenInference. It also includes pre-built integrations for various models and frameworks, making it a holistic solution for managing the lifecycle of LLM applications from development to production.
https://github.com/Agenta-AI/agenta
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
latitude-dev/latitude-llm
Latitude is an open-source AI monitoring platform that provides issue detection, human-aligned evaluations, and agent-native tracing for LLM applications and AI agents.