CuLao - Constructing Utilities of Large Language Models in Resource-Constrained Environments

The increasing development and utilization of Large Language Model (LLM) services have demonstrated many benefits in different contexts. However, LLM services are mainly available in the public cloud and require huge computing resources to operate, thus not accessible to many companies, organizations or communities with constrained resources. While research efforts have concentrated on LLMs quantization for resource-constrained computing environments like edge devices, to democratize the availability of LLM services as utilities for such communities requires much more than the optimization of LLM models. In this paper, we introduce CuLao - a framework for constructing utilities from LLMs in resource-constrained environments. Our framework focuses on key requirements of resource-constrained companies, organizations and communities by enabling the provisioning and coordination of LLMs as utilities, based on the availability of open-source LLMs. CuLao provides techniques and tools for abstracting LLMs as services with suitable APIs and coordinating them as utility ensembles in edge infrastructures.