Awesome Infra for AI › Prompt Management

microsoft/aici

⭐ 2077 Rust repository created 2023-09-26

AICI (Artificial Intelligence Controller Interface) is a framework that allows developers to create and deploy "Controllers" for Large Language Models (LLMs). These controllers are designed to constrain, direct, and modify the output of LLMs token-by-token during inference. By abstracting away the specifics of the underlying LLM inference engine, AICI simplifies the development of advanced control strategies such as constrained decoding, dynamic prompt editing, and coordinating parallel generations. Controllers are implemented as lightweight WebAssembly (Wasm) modules, enabling them to run efficiently alongside the LLM inference engine, utilizing CPU cycles while the GPU handles token generation. This architecture ensures high performance and low overhead. AICI provides flexibility by allowing controllers to be written in any language that compiles to Wasm, operating within a secure, sandboxed environment. It integrates with various LLM inference engines like llama.cpp, HuggingFace Transformers, and custom solutions like rLLM, with vLLM integration planned. The primary goal is to foster innovation in LLM generation control, allowing developers to experiment with and deploy intricate control logic, enhancing the capabilities and reliability of LLM applications. It acts as an abstraction layer, enabling libraries like Guidance and LMQL to leverage its capabilities for improved efficiency and portability across different LLM serving platforms.

https://github.com/microsoft/aici

aiinferencelanguage-modelllmllm-frameworkllm-inferencellm-servingllmopsmodel-servingrusttransformerwasmwasmtimeconstrained-decodingprompt-controlreal-time