Awesome Infra for AI › LLM Gateways & Proxies

apache/apisix

⭐ 17192 Lua repository created 2019-04-10

Apache APISIX is a versatile and high-performance API Gateway designed for the cloud-native era. While its core functionality revolves around traditional API management—offering features like load balancing, dynamic routing, traffic splitting, authentication, observability, and robust security for general microservices—it explicitly extends its capabilities to function as an "AI Gateway." This specialization leverages its flexible plugin system to provide core operational needs for AI and ML models. As an AI Gateway, APISIX enables tasks such as AI proxying, intelligent load balancing specifically tailored for Large Language Models (LLMs), retry mechanisms, and fallback strategies to ensure high availability and resilience of AI services. It also supports token-based rate limiting, crucial for managing access and resource consumption of AI APIs, alongside enhancing the overall security posture for AI agents. The project focuses on the runtime and serving aspects of AI, rather than training or development. Its ability to serve as an ingress controller for Kubernetes also makes it relevant for deploying and managing AI inference workloads in containerized environments. By offering these specific features for AI, APISIX goes beyond a general-purpose gateway, providing purpose-built tools for the operational side of AI.

https://github.com/apache/apisix

ai-gatewayapi-gatewayllm-gatewaymodel-servinginferenceload-balancingapi-managementkubernetestraffic-managementsecurity