Awesome Infra for AI › LLM Gateways & Proxies

Francis1998/nexus-llm-router

⭐ 125 Python added to this list on 2026-07-27 repository created 2026-06-21

Nexus LLM Router is designed for AI infrastructure engineers managing multi-model production pipelines, focusing on optimizing quality, latency, and cost. It acts as middleware, abstracting model choice, fallback mechanisms, budgeting, auditing, and routing logic from the application layer. The router classifies prompt complexity, directing simple tasks to cost-effective models and reserving premium models for intricate queries. It incorporates cost-aware routing, budget guardrails, and Prometheus metrics for spend optimization. Nexus addresses quality variations across different prompt domains by applying deterministic policy rules. It ensures application resilience with per-provider circuit breakers and automatic fallback chains, while also tackling latency spikes through latency-aware routing. The router supports A/B testing of models without code changes and provides durable audit records for compliance. Security features include token-bucket rate limiting, session/tenant budget enforcement, and PII redaction. It offers OpenAI API compatibility, normalizing interactions with various LLM providers like OpenAI, Anthropic, Gemini, and Moonshot, and makes routing policy explicit, testable, and observable. The system features a robust router engine with configurable strategies, an async-first adapter pipeline, comprehensive type safety (mypy compliant), and production readiness with Docker and CI/CD.

https://github.com/Francis1998/nexus-llm-router

ai-agentsai-infrastructurecost-optimizationllm-gatewayllm-routermlopsmulti-llmopenai-compatibleroutingllm-opsproduction-aiai-governanceprompt-routingmodel-selectiondynamic-routingobservabilitysafety-controls