Awesome Infra for AI › LLM Gateways & Proxies

schmitech/orbit

⭐ 352 Python repository created 2025-03-10

ORBIT (Open Retrieval-Based Inference Toolkit) is an AI infrastructure project that enables organizations to deploy secure, private Retrieval-Augmented Generation (RAG) engines and multi-model applications. It functions as an AI gateway and retrieval-adapter layer, allowing users to connect AI models to their business data without sending sensitive information to third-party vendors. The platform supports 100% private and self-hosted operations, compatible with local inference runtimes like Ollama, llama.cpp, or vLLM. Key capabilities include universal data retrievers with out-of-the-box adapters for various databases (PostgreSQL, MongoDB, Elasticsearch), APIs (REST, GraphQL), vector databases, files, and web scraping. It facilitates agentic Model Context Protocol (MCP) tool loops, enabling LLMs to perform multi-step, self-correcting operations within chat sessions. ORBIT also supports cross-adapter skills for dynamic content generation (e.g., text-to-image/video) as part of a conversation workflow. For production environments, it offers a control plane with features like API key validation, rate-limiting, token quotas, content moderation, circuit breakers, and detailed audit logging, ensuring secure inference segregation based on data classification. This flexibility allows dynamic routing of workloads across local, private, or public model infrastructure.

https://github.com/schmitech/orbit

AI gatewayRAGself-hostedmulti-modelprivate AIretrieval-augmented generationinferenceproduction control planesecurityguardrailsLLM orchestrationvector databasesdata retrievalOllamatool-calling