Distributed Offline Load Balancing in MapReduce Networks

Themistoklis Charalambous*, Evangelia Kalyvianaki, Christoforos N. Hadjieostis, Mikael Johansson

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Abstract

In this paper we address the problem of balancing the processing load of MapReduce tasks running on heterogeneous clusters, i. e., clusters composed of nodes with different capacities and update cycles. We present a fully decentralized algorithm, based on ratio consensus, where each mapper decides the amount of workload data to handle for a single user job using only job specific local information, i. e., information that can be collected from directly connected neighboring mappers, regarding their current workload usage and capacity. In contrast to other algorithms in the literature, the proposed algorithm can be deployed in heterogeneous clusters and can operate asynchronously in both directed and undirected communication topologies. The performance of the proposed algorithm is demonstrated via simulation experiments on large-scale strongly connected topologies.

Original languageEnglish
Title of host publication2013 IEEE 52ND ANNUAL CONFERENCE ON DECISION AND CONTROL (CDC)
PublisherIEEE
Pages835-840
Number of pages6
ISBN (Electronic)978-1-4673-5717-3
ISBN (Print)978-1-4673-5714-2
DOIs
Publication statusPublished - 2013
MoE publication typeA4 Article in a conference publication
EventIEEE Conference on Decision and Control - Florence, Italy
Duration: 10 Dec 201313 Dec 2013
Conference number: 52

Publication series

NameIEEE Conference on Decision and Control
PublisherIEEE
ISSN (Print)0743-1546

Conference

ConferenceIEEE Conference on Decision and Control
Abbreviated titleCDC
Country/TerritoryItaly
CityFlorence
Period10/12/201313/12/2013

Fingerprint

Dive into the research topics of 'Distributed Offline Load Balancing in MapReduce Networks'. Together they form a unique fingerprint.

Cite this