Title: MAP task allocation strategy in an ARM-based Hadoop cluster by using local storage as split cache
Authors: Bongen Gu; Yoonsik Kwak
Addresses: Department of Computer Engineering, Korea National University of Transportation, Chungju-Si, 27469, Chungbuk, Republic of Korea ' Department of Computer Engineering, Korea National University of Transportation, Chungju-Si, 27469, Chungbuk, Republic of Korea
Abstract: The increase of power consumption makes the cost of cluster operation higher. One approach for reducing power consumption is to establish a cluster with small nodes which equip a low-power, high-performance processor. Since many low-power consumed nodes do not have storage devices, a separate storage system is required to store large-volume data while nodes mount this storage space to save data. When a Hadoop cluster is configured in such a condition, each node's access to a storage results in excessive network load and delays the execution of Hadoop Map tasks. In this study, we propose a newmap task scheduling policy for Hadoop. This policy transmits multiple splits to nodes at once to reduce network load. In addition, local storage space of nodes is used as a cache for a split, which shortens the time to access splits, so this policy can reduce the execution time of Hadoop applications.
Keywords: Hadoop clusters; ARM; task scheduling; MAP task allocation; local storage; split cache; energy consumption.
International Journal of Advanced Media and Communication, 2016 Vol.6 No.1, pp.65 - 72
Available online: 09 Sep 2016Full-text access for editors Access for subscribers Free access Comment on this article