diegosouzapw/OmniRoute
OmniRoute is a free AI gateway that unifies access to over 170 AI providers, offering token compression, auto-fallback, and aggregating free tiers to provide billions of free tokens monthly.
Awesome Infra for AI › LLM Gateways & Proxies
voidmind-io/voidllm
VoidLLM is a self-hosted LLM proxy and AI gateway that acts as an intermediary between your applications and various LLM providers such as OpenAI, Anthropic, Azure, Ollama, and vLLM. Its core design emphasizes privacy, adhering to a zero-knowledge architecture where prompt or response content is never stored, logged, or persisted. Instead, it tracks only metadata like user, model, token count, and request duration. The gateway provides critical operational features for managing LLM access within an organization. These include: - **Access Control and RBAC**: Virtual keys with granular role-based access control (RBAC) at organizational, team, and user levels. - **Usage Tracking and Cost Management**: Detailed per-key, per-team, and per-organization usage tracking with cost estimation, helping organizations monitor and manage their AI spending. - **Rate Limiting and Budget Enforcement**: Customizable rate limits and token budgets to prevent runaway spending and ensure fair usage across teams. - **Multi-Provider Routing and Load Balancing**: Enables routing requests to different LLM providers, offering model aliases for abstraction, and supports load balancing with options like round-robin, least-latency, and automatic failover for high availability. - **OpenAI Compatibility**: Supports OpenAI-compatible API endpoints for chat completions, embeddings, images, audio, and streaming. - **MCP Gateway**: Functions as an MCP (Model Context Protocol) gateway, allowing for the proxying of external MCP servers and providing built-in tools for model listing, health checks, and usage reporting. It also includes a "Code Mode" for WASM-sandboxed JavaScript to orchestrate multi-tool calls. VoidLLM is deployable via Docker, Helm charts, or as a standalone binary, offering Prometheus metrics for observability and supporting SQLite or PostgreSQL for data storage.
https://github.com/voidmind-io/voidllm
Also in LLM Gateways & Proxies
Mintplex-Labs/anything-llm
AnythingLLM is an all-in-one local-first AI application for chatting with documents, managing AI agents, and integrating with various LLMs and vector databases, offering dynamic model routing and m...
BerriAI/litellm
LiteLLM is an open-source AI Gateway and Python SDK providing a unified interface to over 100 LLM providers, with features like cost tracking, guardrails, load balancing, and observability.
QuantumNous/new-api
new-api is a unified LLM gateway and AI asset management system that enables aggregation, distribution, and cross-conversion of various LLMs into OpenAI, Claude, or Gemini compatible formats, offer...
Kong/kong
Kong Gateway is a cloud-native API and AI gateway offering high performance, extensibility via plugins, and advanced AI traffic capabilities including multi-LLM support, semantic security, and cach...
decolua/9router
9Router is an AI router and token saver that connects various AI coding tools to over 40 AI providers, optimizing usage with auto-fallback, quota tracking, and token compression.
apache/apisix
Apache APISIX is a dynamic, real-time, high-performance API Gateway that can also function as an AI Gateway, providing AI proxying, load balancing for LLMs, and robust security for AI agents.
lidge-jun/opencodex
OpenCodex is a universal local proxy that enables OpenAI Codex, Claude Code, and Grok Build to utilize any LLM provider, offering advanced model routing, account pooling, and API key management.