langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
palico-ai/palico-ai
Palico AI provides a comprehensive tech-stack for developers to build, iterate on, and deploy LLM applications. It emphasizes an iterative development cycle, allowing users to experiment with different models, prompts, RAG datasets, and custom code. Key features include a flexible application building environment, local preview with a Playground UI, and mechanisms for performance improvement through experiments and evaluations. The framework integrates robust debugging capabilities with comprehensive logs and traces, crucial for identifying and resolving issues in LLM workflows. It supports seamless integration with various frontend applications via Client SDKs or REST APIs, and offers continuous integration (CI) and pull-request preview setups for streamlined deployment. Palico AI also includes a control panel for application management. It allows for detailed control over LLM application implementation, supporting complex interactions such as streaming, memory management for conversation states, and tool execution for agentic behaviors. Feature flags enable dynamic swapping of models, prompts, RAG configurations, and custom logic at runtime. Palico AI is designed to be compatible with popular LLM tools and libraries including Langchain, LlamaIndex, OpenAI, Anthropic, and various vector databases like Pinecone, PG Vector, and Chroma, ensuring flexibility and broad applicability within the LLM development ecosystem.
https://github.com/palico-ai/palico-ai
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.