Awesome Infra for AI › LLM Gateways & Proxies

mozilla-ai/otari

⭐ 504 Python added to this list on 2026-07-13 repository created 2026-04-02

Otari is an open-source, self-hostable LLM gateway designed to simplify the management and operation of large language models across multiple providers. It offers a single OpenAI-compatible endpoint that can route requests to more than 40 different LLM providers through its `any-llm` SDK. Key functionalities include robust authentication and authorization with virtual API keys, allowing granular control over access and revocation. A core feature is its cost control mechanism, which enforces per-user and per-key budgets *before* requests are processed, preventing unexpected overspending. Additionally, Otari meticulously tracks all usage and spend data across various models and applications, providing queryable logs. This allows organizations to maintain control over their LLM infrastructure, keeping provider credentials and sensitive usage data within their own environment. It natively supports a standalone mode with SQLite and offers a full stack deployment via Docker Compose for more robust, durable setups. Otari is part of a larger ecosystem that includes client SDKs for Python, TypeScript, Rust, and Go, as well as a CLI tool, making it a comprehensive solution for managing LLM inference in production.

https://github.com/mozilla-ai/otari

ai-gatewayllm-proxyllm-routingapi-key-managementcost-trackingbudgetsusage-trackingmulti-provideropenai-compatibleself-hostedllmopsguardrails