Awesome Infra for AI › LLM Gateways & Proxies

vllm-project/semantic-router

⭐ 6031 Go repository created 2025-08-26

vLLM Semantic Router serves as a signal-driven intelligent router developed to address the complexities of managing and orchestrating multiple AI/ML models in the LLM era. Its primary purpose is to help organizations build efficient, safer, and more adaptive model systems that can operate across cloud, data center, and edge environments. This tool is crucial for optimizing the use of various models that differ in capability, scale, cost, and privacy considerations. It provides core values by enhancing token economics through reducing wasted tokens and maximizing effective output, ensuring LLM safety by detecting jailbreaks, sensitive data leakage, and hallucinations, and enabling fullmesh intelligence through the coordination of local, private, and frontier models. The Semantic Router aims to provide a robust infrastructure layer for intelligently directing prompts and tasks to the most suitable AI models, thereby improving overall system performance and cost-effectiveness. It is particularly relevant for applications requiring dynamic model selection, such as those involving complex AI agents or multi-modal systems, and integrates with the vLLM ecosystem for high-performance inference.

https://github.com/vllm-project/semantic-router

intelligent routermixture-of-modelsLLM gatewayAI safetytoken economicsinference optimizationsemantic routingLLM orchestrationmodel serving