Awesome Infra for AI › LLM Gateways & Proxies

intentee/paddler

⭐ 1677 Rust repository created 2024-04-27

Paddler is an open-source, self-contained platform designed for deploying, serving, and scaling Large Language Models (LLMs) and Vision-Language Models (VLMs) on private infrastructure. It functions as an LLM load balancer, primarily utilizing a built-in llama.cpp engine for inference, supporting both CPU and GPU operations. The platform aims to provide privacy, reliability, cost control, and independence from closed-source model providers, making it suitable for product teams needing LLM inference, DevOps/LLMOps teams scaling models, and organizations with strict compliance or privacy requirements. Key features include LLM-specific load balancing, dynamic agent integration with autoscaling capabilities, request buffering for scaling from zero, and dynamic model swapping. It offers a web admin panel for management, monitoring, and testing, as well as observability metrics. Paddler consists of two main components: a 'balancer' that distributes incoming requests and exposes inference, management, and web admin services; and 'agents' that generate tokens and embeddings through dedicated slots. The project also provides a desktop application for more casual local AI cluster setups. It is designed for straightforward installation as a single binary and focuses on providing an efficient, self-hosted solution for production LLM inference.

https://github.com/intentee/paddler

aillamacppllmllmopsload-balancermodel servinginferenceenterprise llmself-hosted ai