langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
deepsense-ai/ragbits
Ragbits is a comprehensive set of building blocks designed to accelerate the development, deployment, and operation of Generative AI applications. It offers modular components for various aspects of GenAI lifecycle, starting with flexible LLM integration that allows switching between over 100 LLMs (via LiteLLM) or local models, ensuring type-safe LLM calls for robust interactions. For Retrieval-Augmented Generation (RAG), Ragbits supports ingesting data from 20+ formats and various sources (S3, GCS, Azure) with scalable, distributed processing, and connectivity to popular vector stores like Qdrant and PgVector. The framework also facilitates the creation of multi-agent workflows through an Agent-to-Agent (A2A) protocol and Model Context Protocol (MCP) for real-time data integration. Operational aspects are well-covered with real-time observability via OpenTelemetry, built-in testing integrations like promptfoo for prompt validation, and continuous optimization features. It also provides utilities for AI safety and guardrails, ensuring responses are safe and relevant. Ragbits modular architecture allows users to install only the necessary components, such as `ragbits-core` for fundamental LLM and vector database tools, `ragbits-agents` for agentic systems, `ragbits-document-search` for RAG pipelines, `ragbits-evaluate` for testing, `ragbits-guardrails` for safety, and `ragbits-chat` for conversational AI infrastructure. This makes it a versatile tool for building, deploying, and managing production-ready GenAI applications.
https://github.com/deepsense-ai/ragbits
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.