Title: MAP task allocation strategy in an ARM-based Hadoop cluster by using local storage as split cache

Authors: Bongen Gu; Yoonsik Kwak

Addresses: Department of Computer Engineering, Korea National University of Transportation, Chungju-Si, 27469, Chungbuk, Republic of Korea ' Department of Computer Engineering, Korea National University of Transportation, Chungju-Si, 27469, Chungbuk, Republic of Korea

Abstract: The increase of power consumption makes the cost of cluster operation higher. One approach for reducing power consumption is to establish a cluster with small nodes which equip a low-power, high-performance processor. Since many low-power consumed nodes do not have storage devices, a separate storage system is required to store large-volume data while nodes mount this storage space to save data. When a Hadoop cluster is configured in such a condition, each node's access to a storage results in excessive network load and delays the execution of Hadoop Map tasks. In this study, we propose a newmap task scheduling policy for Hadoop. This policy transmits multiple splits to nodes at once to reduce network load. In addition, local storage space of nodes is used as a cache for a split, which shortens the time to access splits, so this policy can reduce the execution time of Hadoop applications.

Keywords: Hadoop clusters; ARM; task scheduling; MAP task allocation; local storage; split cache; energy consumption.

DOI: 10.1504/IJAMC.2016.079108

International Journal of Advanced Media and Communication, 2016 Vol.6 No.1, pp.65 - 72

Accepted: 26 May 2016
Published online: 12 Sep 2016 *