Title: Improving execution speed of incremental runs of MapReduce using provenance

Authors: Anu Mary Chacko; Anish Gupta; S. Madhu; S.D. Madhu Kumar

Addresses: Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India ' Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India ' Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India ' Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India

Abstract: Hadoop MapReduce is an analytic tool used to solve big data problems that are parallelisable. MapReduce jobs need to be rerun frequently for data changes. Many times these data changes are made by appending new data to the existing file. So in a rerun, if we can reuse the output of the previous run and limit the job execution to the new data, we can reduce the overall job execution time. In the literature, there are schemes that use the concept of memoisation, storing the intermediate result, etc. to implement efficient incremental rerun. In this paper, we explain how provenance can be used to implement transparent incremental MapReduce for 'append only' input files. Our approach requires no additional storage or modification of existing Hadoop infrastructure or scheduler. Experimental evaluation of running MapReduce on multinode cluster with provenance stored in HBase gave good results for incremental runs in cases of an addition of new file/new data.

Keywords: Hadoop MapReduce; provenance; HBase; incremental MapReduce.

DOI: 10.1504/IJBDI.2017.085521

International Journal of Big Data Intelligence, 2017 Vol.4 No.3, pp.186 - 194

Received: 05 Aug 2015
Accepted: 21 Apr 2016
Published online: 30 Jul 2017 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article