Title: A rack-aware scalable resource management system for Hadoop YARN

Authors: Timothy Moses; Hyacinth C. Inyiama; Sylvanus O. Anigbogu

Addresses: Department of Computer Science, Federal University of Lafia, Nasarawa State, Nigeria ' Department of Electronic and Computer Engineering, Nnamdi Azikiwe University, Awka, Anambra State, Nigeria ' Department of Computer Science, Nnamdi Azikiwe University, Awka, Anambra State, Nigeria

Abstract: Big data have brought in an era of data exploration and utilisation with MapReduce computational paradigm as its major enabler. Though great efforts through the implementation of Hadoop have made computation scale to tens of thousands of commodity cluster processors, the centralised architecture of resource manager has adversely affected response time in large data centres. The developed model decouples the responsibilities of resource manager by providing another layer where each daemon called rack unit resource manager (RU_RM) carries out the responsibility of allocating resources to compute nodes within its local rack to ensure low latency for large files. The application was developed and tested with Hadoop workload benchmarks used for analysis. Two performance evaluation metrics (efficiency and average task-delay ratio) were used for comparison. Efficiency quantifies average cluster utilisation while average task-delay ratio measures average delay time. Results obtained showed that as file size increases, the developed model outperforms the existing framework.

Keywords: MapReduce; Hadoop; framework; scalable; rack-aware; resource manager; big data; rack unit resource manager.

DOI: 10.1504/IJHPCN.2020.110257

International Journal of High Performance Computing and Networking, 2020 Vol.16 No.1, pp.1 - 13

Received: 23 May 2019
Accepted: 24 Mar 2020
Published online: 12 Oct 2020 *