Title: Multi-source streaming-based data accesses for MapReduce systems

Authors: Jiadong Wu; Bo Hong

Addresses: School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA ' School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

Abstract: The MapReduce programming model, along with its open-source implementation - Hadoop - has provided a cost effective solution for many data intensive applications. Hadoop stores data distributively and exploits data locality by assigning tasks to where data is stored. In many cases, however, accessing remote data (rack-local and off-rack) is inevitable. In this paper we are evaluating the possibility of improving the remote data accessing performance by streaming data from multiple available replicas. The proposed design consists of a circular buffer, a slice reader and an enhanced DataNode. Such system is capable of adapting to both the static performance variance caused by network topology as well as dynamic variance caused by congestion. Extensive experiments show that multisource streaming can significantly improve the throughput of remote data access and accelerate the related map tasks by 10%-20%. For systems with heterogenous network links, upto 4× speedup was observed.

Keywords: MapReduce; multisource streaming; remote data access; network topology; congestion; big data.

DOI: 10.1504/IJBDI.2014.063843

International Journal of Big Data Intelligence, 2014 Vol.1 No.1/2, pp.36 - 49

Received: 10 Jan 2014
Accepted: 02 Apr 2014

Published online: 30 Sep 2014 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article