Queueing Analysis of an Ensemble Machine Learning System

Recent advances in AI/ML technologies have accelerated the development of various ML applications. One of the major trends in AI/ML application development is the increasing use of multiple ML models to support high-accuracy inference in a complex end-to-end ML serving. However, testing the right configuration of multiple ML models is expensive, and the application requirements for ML inferences are highly dependent on various factors like the quality of ML models, computing resource performance, and data quality. In this context, techniques and methods that help to emulate and analyze ML inference characteristics using queueing theory can reduce the development effort and cost for ML services encapsulating ML models but also the entire ML system. In this paper, we modeled and analyzed a queueing model for an ML system that uses ensemble learning as an inference method with a new rule and clarified the impacts of model design in ensemble learning on the system’s performance. As a result, we demonstrate the usefulness of the analysis for understanding possible configurations and their efficiency in the ML system through queueing analysis and simulation.