Awesome Infra for AI › LLM Gateways & Proxies

adaline/gateway

⭐ 608 TypeScript repository created 2024-10-15

Adaline Gateway is a 'Super SDK' designed to simplify and standardize interactions with a vast array of Large Language Models (LLMs). Unlike a proxy, it's a fully local, TypeScript-based solution that can be deployed anywhere, ensuring privacy and control. The gateway abstracts away the complexities of different LLM providers, offering a single, powerful API for functions like chat completion and embedding generation across numerous models from vendors such as OpenAI, Anthropic, Google, AWS Bedrock, and more. Key operational features include automatic batching for requests, resilient retries with exponential backoff, and configurable caching to optimize performance and cost. For enhanced oversight and integration into existing infrastructure, it provides comprehensive callback mechanisms and out-of-the-box OpenTelemetry support for tracing and observability. Its extensible architecture allows for custom plugins for caching, logging, and HTTP clients, making it adaptable to various production environments. The SDK also supports plug-and-play custom providers, enabling integration with local or bespoke models. This focus on abstracting LLM interaction, coupled with operational features like caching, retries, and observability, positions it as a critical tool for managing and serving AI inference workloads.

https://github.com/adaline/gateway

LLMgatewayproxyinferenceservingobservabilitycachingretriesOpenTelemetrymulti-providerTypeScriptSDKbatching