Awesome Infra for AI › LLM Gateways & Proxies

peva3/SmarterRouter

⭐ 153 Python repository created 2026-02-16

SmarterRouter acts as an intelligent LLM gateway and router, sitting between applications and various LLM providers (Ollama, llama.cpp, OpenAI, Anthropic, etc.). Its core purpose is to automate the selection of the optimal LLM for a given prompt, eliminating the need for manual configuration. It achieves this by profiling available models on local hardware to understand their performance characteristics and by using benchmark data for external models. The router is VRAM-aware, monitoring GPU usage across NVIDIA, AMD, Intel, and Apple Silicon, and dynamically managing model loading and unloading. Key features include intelligent routing, semantic caching for instant responses to identical prompts, automatic failover to backup models, and continuous learning from user feedback to refine routing decisions. It offers a local-first approach, reducing reliance on cloud APIs, and is OpenAI-compatible for seamless integration with existing clients. SmarterRouter comes with production-ready features like monitoring, metrics, and an interactive setup wizard for easy deployment via Docker. It supports dynamic addition/removal of models, enhanced cache analytics, and performance optimizations through async GPU I/O and batched queries, aiming to provide a robust and efficient solution for operating LLM inference in self-hosted and hybrid environments.

https://github.com/peva3/SmarterRouter

ai-cacheai-gatewaydockerfastapigpu-monitoringllmllm-proxyllm-routerlocal-llmmodel-servingollamaollama-apiopenai-proxyself-hostedself-hosted-aisemantic-cache