Authors: Jiadong Wu; Bo Hong
Addresses: School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA ' School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA
Abstract: The MapReduce programming model, along with its open-source implementation - Hadoop - has provided a cost effective solution for many data intensive applications. Hadoop stores data distributively and exploits data locality by assigning tasks to where data is stored. In many cases, however, accessing remote data (rack-local and off-rack) is inevitable. In this paper we are evaluating the possibility of improving the remote data accessing performance by streaming data from multiple available replicas. The proposed design consists of a circular buffer, a slice reader and an enhanced DataNode. Such system is capable of adapting to both the static performance variance caused by network topology as well as dynamic variance caused by congestion. Extensive experiments show that multisource streaming can significantly improve the throughput of remote data access and accelerate the related map tasks by 10%-20%. For systems with heterogenous network links, upto 4× speedup was observed.
Keywords: MapReduce; multisource streaming; remote data access; network topology; congestion; big data.
International Journal of Big Data Intelligence, 2014 Vol.1 No.1/2, pp.36 - 49
Available online: 23 Jul 2014Full-text access for editors Access for subscribers Purchase this article Comment on this article