Awesome Infra for AI › LLM Gateways & Proxies

ferro-labs/ai-gateway

⭐ 271 Go repository created 2026-02-25

Ferro Labs AI Gateway is an open-source, high-performance Go-native solution designed for routing and managing LLM requests across a multitude of AI providers, including OpenAI, Anthropic, Bedrock, and Azure. It aims to offer a unified OpenAI-compatible API for over 30 LLM providers and 2,500+ models. The gateway is built for efficiency and scalability, boasting capabilities like a single binary deployment, minimal memory footprint (32MB base), and high request throughput (benchmarked at 13,925 RPS with 1,000 concurrent users). Key features include 8 routing strategies (fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional), configurable provider failover, and per-request model aliases. Beyond routing, it incorporates essential operational features such as response caching, guardrails (for PII detection, content moderation), prompt management, and cost controls to optimize LLM usage. It is positioned as an performant alternative to Python-based proxies like LiteLLM and other solutions, emphasizing its Go-native architecture for predictable latency and minimal resource consumption. The project targets developers and organizations looking for a robust, efficient, and feature-rich solution to manage their production AI inference workloads.

https://github.com/ferro-labs/ai-gateway

ai-gatewayai-infrastructuregatewayguardrailskonglitellmllmllm-costllm-proxyllm-strategyllmopsmcppii-detectionprompt-managementsemantic-cache