Int. J. of Big Data Intelligence   »   2017 Vol.4, No.3

 

 

Title: Improving execution speed of incremental runs of MapReduce using provenance

 

Authors: Anu Mary Chacko; Anish Gupta; S. Madhu; S.D. Madhu Kumar

 

Addresses:
Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India
Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India
Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India
Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India

 

Abstract: Hadoop MapReduce is an analytic tool used to solve big data problems that are parallelisable. MapReduce jobs need to be rerun frequently for data changes. Many times these data changes are made by appending new data to the existing file. So in a rerun, if we can reuse the output of the previous run and limit the job execution to the new data, we can reduce the overall job execution time. In the literature, there are schemes that use the concept of memoisation, storing the intermediate result, etc. to implement efficient incremental rerun. In this paper, we explain how provenance can be used to implement transparent incremental MapReduce for 'append only' input files. Our approach requires no additional storage or modification of existing Hadoop infrastructure or scheduler. Experimental evaluation of running MapReduce on multinode cluster with provenance stored in HBase gave good results for incremental runs in cases of an addition of new file/new data.

 

Keywords: Hadoop MapReduce; provenance; HBase; incremental MapReduce.

 

DOI: 10.1504/IJBDI.2017.10006111

 

Int. J. of Big Data Intelligence, 2017 Vol.4, No.3, pp.186 - 194

 

Available online: 28 Jul 2017

 

 

Editors Full text accessAccess for SubscribersPurchase this articleComment on this article