Awesome Infra for AI › LLM Gateways & Proxies

thushan/olla

⭐ 312 Go repository created 2025-05-23

Olla is an efficient and high-performance proxy and load balancer specifically designed for LLM inference infrastructure. It enables intelligent routing of LLM requests across various self-hosted inference nodes, offering automatic failover and connection retry for enhanced reliability. The tool supports unified model discovery and cataloging across different providers, allowing seamless routing to available models on compatible endpoints. Olla integrates with a wide range of popular LLM inference backends such as Ollama, LM Studio, vLLM, and llama.cpp, and is designed to complement existing API gateways and orchestration platforms. It features advanced capabilities including priority-based routing, KV-cache aware sticky sessions for multi-turn conversations, smart model unification with OpenAI-compatible cross-provider routing, and dual proxy engines (Sherpa for simplicity, Olla for high performance). Health monitoring with circuit breakers, intelligent retries, self-healing model discovery, and detailed request tracking are also integral features. Olla is built for production environments, offering rate limiting, request size limits, configurable timeouts, and efficient resource usage (under 50MB RAM). It also includes Anthropic Messages API passthrough or translation and backend authentication for secure inference.

https://github.com/thushan/olla

llm-proxyllm-routerload-balancerinference-managementllm-inferenceself-hosted-aifailovermodel-discoveryhigh-performancegolangvllmollamallama.cpplmstudiosglangmlx