Awesome Infra for AI › LLM Gateways & Proxies

atopos31/llmio

⭐ 317 TypeScript repository created 2025-08-06

LLMIO is a robust, Go-based LLM load-balancing gateway designed to streamline the integration and management of multiple large language models (LLMs) from different providers. It offers a unified REST API compatible with OpenAI Chat Completions, Anthropic Messages, and Gemini Native, supporting both streaming and non-streaming passthrough. Key features include weighted scheduling with two strategies (random by weight, priority by weight) for intelligent request routing based on tool calling, structured output, and multimodal capabilities. The platform boasts an intuitive Admin Web UI built with React, TypeScript, Tailwind, and Vite, enabling easy configuration of providers, models, associations, and real-time monitoring of logs and metrics. LLMIO also incorporates essential operational features such as built-in rate limiting with fallback mechanisms, provider connectivity checks for fault isolation, and local persistence for configurations and request logs using SQLite. A standout feature is its comprehensive observability, tracking every request with TraceID, detailed latency breakdowns (proxy, first-chunk, completion time), TPS, and token usage (input, cached, output). It calculates per-request costs based on configurable per-million-token prices, offering deep insights into LLM usage and expenditure. Additionally, it supports session tracking, allowing users to tag and filter logs by session ID for enhanced debugging and analysis.

https://github.com/atopos31/llmio

LLM gatewayload balancingobservabilitycost trackingOpenAIAnthropicGeminiAPI proxyLLM operationsGoGolang