Title: Load balancing in MapReduce on homogeneous and heterogeneous clusters: an in-depth review

Authors: Mohammad Javad Kargar; Meysam Vakili

Addresses: Department of Computer Engineering, College of Engineering, University of Science and Culture, Tehran, Iran ' Department of Computer Engineering, College of Engineering, University of Science and Culture, Tehran, Iran

Abstract: Numbers of various programming models have been proposed to process big data in recent years. However, MapReduce is the most famous programming model amongst cloud computing environments and includes many advantages, yet there are several challenges to deal with. Load balancing is considered as one of the most significant downsides of MapReduce which causes the increase in applications' runtime and accordingly results in less-efficiency, where there is no appropriate proposed mechanism. Although, data locality and data skew are known as two main key factors for determination of load balancing, yet it is remarkable that load balance highly depends on whether the computational clusters are homogeneous or heterogeneous. This paper examines the effectiveness of two main key factors. These are data locality and data skew on homogeneous and heterogeneous clusters. Besides, a review is conducted on a number of recent literature in the same context of load balancing improvements in Hadoop MapReduce. Finally, all investigated researches are compared with the purpose of highlighting the differences of various load balancing methods, the optimisation phase, type of clusters and the main challenges.

Keywords: MapReduce; Hadoop; load balancing; big data; homogeneous clusters; heterogeneous clusters; cloud computing; data locality; data skew; optimisation.

DOI: 10.1504/IJCNDS.2015.070969

International Journal of Communication Networks and Distributed Systems, 2015 Vol.15 No.2/3, pp.149 - 168

Received: 11 Jul 2014
Accepted: 26 Dec 2014
Published online: 04 Aug 2015 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article