diegosouzapw/OmniRoute
OmniRoute is a free AI gateway that unifies access to over 170 AI providers, offering token compression, auto-fallback, and aggregating free tiers to provide billions of free tokens monthly.
Awesome Infra for AI › LLM Gateways & Proxies
BerriAI/litellm
LiteLLM serves as an open-source AI Gateway and Python SDK, designed to simplify interactions with a multitude of Large Language Model (LLM) providers, including OpenAI, Anthropic, Gemini, Bedrock, and Azure. It provides a single, unified interface, allowing developers to call over 100 different LLMs using the familiar OpenAI format, abstracting away the complexities of disparate SDKs, authentication patterns, request formats, and error types. This enables seamless switching between LLM providers without extensive code rewrites. The project offers two primary modes of operation: a Python SDK for direct library integration into applications and an AI Gateway (Proxy Server) that can be deployed as a centralized service. The gateway functionality includes essential production-ready features such as virtual keys for access control, comprehensive spend tracking, customizable guardrails for safety and governance, intelligent load balancing across multiple LLM endpoints, and an administrative dashboard for monitoring and management. It boasts low latency, with P95 latency reported at 8ms at 1k RPS. LiteLLM also supports Agents-to-Agents (A2A) protocols for invoking various AI agents and integrates with Multi-Cloud Provider (MCP) servers. Its core value lies in streamlining LLM operations, making it easier for teams to build and manage applications that leverage diverse AI models in a production environment.
https://github.com/BerriAI/litellm
Also in LLM Gateways & Proxies
Mintplex-Labs/anything-llm
AnythingLLM is an all-in-one local-first AI application for chatting with documents, managing AI agents, and integrating with various LLMs and vector databases, offering dynamic model routing and m...
QuantumNous/new-api
new-api is a unified LLM gateway and AI asset management system that enables aggregation, distribution, and cross-conversion of various LLMs into OpenAI, Claude, or Gemini compatible formats, offer...
Kong/kong
Kong Gateway is a cloud-native API and AI gateway offering high performance, extensibility via plugins, and advanced AI traffic capabilities including multi-LLM support, semantic security, and cach...
decolua/9router
9Router is an AI router and token saver that connects various AI coding tools to over 40 AI providers, optimizing usage with auto-fallback, quota tracking, and token compression.
apache/apisix
Apache APISIX is a dynamic, real-time, high-performance API Gateway that can also function as an AI Gateway, providing AI proxying, load balancing for LLMs, and robust security for AI agents.
lidge-jun/opencodex
OpenCodex is a universal local proxy that enables OpenAI Codex, Claude Code, and Grok Build to utilize any LLM provider, offering advanced model routing, account pooling, and API key management.
casdoor/casdoor
Casdoor is an open-source, "AI-first" Identity and Access Management (IAM) and Model Context Protocol (MCP) gateway, providing authentication and authorization for AI applications and agents.