CuLao - Constructing Utilities of Large Language Models in Resource-Constrained Environments

Linh Truong, Nguyen Ngoc Nhu Trang

Tutkimustuotos: TyöpaperiEsipainosScientific

15 Lataukset (Pure)


The increasing development and utilization of Large Language Model (LLM) services have demonstrated many benefits in different contexts. However, LLM services are mainly available in the public cloud and require huge computing resources to operate, thus not accessible to many companies, organizations or communities with constrained resources. While research efforts have concentrated on LLMs quantization for resource-constrained computing environments like edge devices, to democratize the availability of LLM services as utilities for such communities requires much more than the optimization of LLM models. In this paper, we introduce CuLao - a framework for constructing utilities from LLMs in resource-constrained environments. Our framework focuses on key requirements of resource-constrained companies, organizations and communities by enabling the provisioning and coordination of LLMs as utilities, based on the availability of open-source LLMs. CuLao provides techniques and tools for abstracting LLMs as services with suitable APIs and coordinating them as utility ensembles in edge infrastructures.
TilaJätetty - 6 tammik. 2024
OKM-julkaisutyyppiEi oikeutettu


Sukella tutkimusaiheisiin 'CuLao - Constructing Utilities of Large Language Models in Resource-Constrained Environments'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä