Awesome Infra for AI › LLM Gateways & Proxies

Chleba/ollamaMQ

⭐ 127 Rust repository created 2026-02-27

`ollamaMQ` is a robust and high-performance asynchronous message queue dispatcher and load balancer designed specifically for managing inference requests to Ollama and LM Studio API instances. Acting as a smart proxy, it efficiently distributes incoming requests from multiple users across several backend models. Its core features include multi-backend load balancing using a combination of least-connections and round-robin strategies, automatically detecting and routing requests to compatible Ollama or OpenAI-compatible backends. The proxy also implements model-aware routing, ensuring requests are sent only to instances that have the required model loaded, preventing errors and optimizing resource usage. Key capabilities include parallel processing of requests to maximize throughput, continuous backend health checks to maintain reliability, and a sophisticated per-user queuing system with fair-share scheduling to prevent resource monopolization. It supports VIP and Boost modes for priority users and provides transparent header forwarding. A real-time Text-based User Interface (TUI) dashboard offers live monitoring of backend health, active requests, queue depths, and overall throughput. Built in Rust with `tokio` and `axum`, it offers an efficient and concurrent solution for managing LLM inference workloads, supporting various Ollama and OpenAI-compatible API endpoints.

https://github.com/Chleba/ollamaMQ

ollamaproxyllm gatewayload balancingfair-sharequeuingtuirustopenai-compatiblemodel-aware routinginference serving