langfuse/langfuse
Langfuse is an open-source LLM engineering platform for developing, monitoring, evaluating, and debugging AI applications, offering observability, prompt management, and evaluation capabilities.
Awesome Infra for AI › LLM Observability & Tracing
inferock/inferock-bench
inferock-bench operates as a local diagnostic proxy for metered API traffic to various LLMs, including OpenAI, Anthropic, Gemini, and select OpenRouter endpoints. Its primary function is to provide an independent, per-call receipt for AI/LLM API interactions. It monitors and records provider-reported usage, pricing evidence, request/response metadata, status, timing, and retry evidence. Crucially, it detects and logs delivery failures that can impact billing accuracy, such as billed-empty output, refusals, truncations, token-recount mismatches, duplicate request IDs, and cache-discount-at-risk signals. The tool generates receipts using the `@inferock/measure` grading code and "The Inferock Standard," differentiating between observed provider spend, bill-bounded money loss, time loss, and invoice-check exposure. It emphasizes local operation, ensuring that provider keys are not sent to Inferock and receipts remain on the user's machine unless explicitly shared. This allows users to audit their AI bills, verify token usage locally, and identify situations where they might have been incorrectly charged for failed API calls, thereby enhancing transparency and accountability in LLM operational costs.
https://github.com/inferock/inferock-bench
Also in LLM Observability & Tracing
comet-ml/opik
Opik is an open-source platform for comprehensive observability, evaluation, and optimization of LLM applications, RAG systems, and agentic workflows.
raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK for comprehensive observability, monitoring, and evaluation of AI agents and LLM applications, offering tracing, debugging, and advanced analytics.
Arize-ai/phoenix
Phoenix is an open-source AI observability platform for LLM application experimentation, evaluation, and troubleshooting, providing tracing, evaluation, dataset management, prompt management, and a...
VoltAgent/voltagent
VoltAgent is an end-to-end AI Agent Engineering Platform offering an open-source TypeScript framework for building intelligent agents and a VoltOps Console for observability, automation, deployment...
traceloop/openllmetry
OpenLLMetry provides open-source observability for LLM applications by extending OpenTelemetry to capture traces and metrics from LLM providers, vector databases, and AI frameworks.
Helicone/helicone
Helicone is an open-source LLM observability platform and AI gateway that provides monitoring, evaluation, prompt management, and intelligent routing for large language models.
Agenta-AI/agenta
Agenta is an open-source LLMOps platform designed to accelerate the development of reliable LLM applications, offering integrated prompt management, evaluation, and observability features.