Int. J. of Big Data Intelligence   »   2014 Vol.1, No.1/2

 

 

Title: Multi-source streaming-based data accesses for MapReduce systems

 

Authors: Jiadong Wu; Bo Hong

 

Addresses:
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

 

Abstract: The MapReduce programming model, along with its open-source implementation - Hadoop - has provided a cost effective solution for many data intensive applications. Hadoop stores data distributively and exploits data locality by assigning tasks to where data is stored. In many cases, however, accessing remote data (rack-local and off-rack) is inevitable. In this paper we are evaluating the possibility of improving the remote data accessing performance by streaming data from multiple available replicas. The proposed design consists of a circular buffer, a slice reader and an enhanced DataNode. Such system is capable of adapting to both the static performance variance caused by network topology as well as dynamic variance caused by congestion. Extensive experiments show that multisource streaming can significantly improve the throughput of remote data access and accelerate the related map tasks by 10%-20%. For systems with heterogenous network links, upto 4× speedup was observed.

 

Keywords: MapReduce; multisource streaming; remote data access; network topology; congestion; big data.

 

DOI: 10.1504/IJBDI.2014.063843

 

Int. J. of Big Data Intelligence, 2014 Vol.1, No.1/2, pp.36 - 49

 

Submission date: 07 Jan 2014
Date of acceptance: 02 Apr 2014
Available online: 23 Jul 2014

 

 

Editors Full text accessAccess for SubscribersPurchase this articleComment on this article