Awesome Infra for AI › LLM Gateways & Proxies

TPIsoftwareOSPO/digiRunner-Open-Source

⭐ 307 Java repository created 2024-09-30

digiRunner is an API Gateway designed to manage and orchestrate both traditional microservices and Large Language Models (LLMs). It positions itself as an "AI Gateway" that addresses common challenges in adopting generative AI, such as vendor lock-in, unpredictable costs, and security risks. Key AI-specific features include provider abstraction to switch backend models without code changes, secure API key management preventing exposure in client-side code, and granular cost control through FinOps and Tokenomics-aware policies. It enables enforcing input and output limits for LLMs to manage costs effectively. Furthermore, digiRunner provides a prompt engineering capability by offering an AI Prompt Template engine for creating, managing, and standardizing prompts, treating them as versioned infrastructure. While it maintains core API management functionalities for RESTful services like high performance, Kubernetes integration, and full lifecycle management, its unique value proposition lies in its specialized features for governing and securing AI service consumption, making it a critical component for enterprises integrating AI into their operations.

https://github.com/TPIsoftwareOSPO/digiRunner-Open-Source

ai-gatewayapi-gatewayllm-gatewayprompt-managementai-governancecost-controlapi-managementmicroserviceskubernetessecurity