Int. J. of Advanced Media and Communication   »   2016 Vol.6, No.1

 

 

You can view the full text of this article for Free access using the link below.

 

 

Title: MAP task allocation strategy in an ARM-based Hadoop cluster by using local storage as split cache

 

Authors: Bongen Gu; Yoonsik Kwak

 

Addresses:
Department of Computer Engineering, Korea National University of Transportation, Chungju-Si, 27469, Chungbuk, Republic of Korea
Department of Computer Engineering, Korea National University of Transportation, Chungju-Si, 27469, Chungbuk, Republic of Korea

 

Abstract: The increase of power consumption makes the cost of cluster operation higher. One approach for reducing power consumption is to establish a cluster with small nodes which equip a low-power, high-performance processor. Since many low-power consumed nodes do not have storage devices, a separate storage system is required to store large-volume data while nodes mount this storage space to save data. When a Hadoop cluster is configured in such a condition, each node's access to a storage results in excessive network load and delays the execution of Hadoop Map tasks. In this study, we propose a newmap task scheduling policy for Hadoop. This policy transmits multiple splits to nodes at once to reduce network load. In addition, local storage space of nodes is used as a cache for a split, which shortens the time to access splits, so this policy can reduce the execution time of Hadoop applications.

 

Keywords: Hadoop clusters; ARM; task scheduling; MAP task allocation; local storage; split cache; energy consumption.

 

DOI: 10.1504/IJAMC.2016.079108

 

Int. J. of Advanced Media and Communication, 2016 Vol.6, No.1, pp.65 - 72

 

Date of acceptance: 26 May 2016
Available online: 09 Sep 2016

 

 

Editors Full text accessFree access Free accessComment on this article