Int. J. of Big Data Intelligence   »   2015 Vol.2, No.2

 

 

Title: Heterogeneity-aware scheduler for stream processing frameworks

 

Authors: Marek Rychlý; Petr Škoda; Pavel Smrž

 

Addresses:
Department of Information Systems, Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic
Department of Computer Graphics and Multimedia, Faculty of Information Technology, Brno University of Technology, IT4Innovations Centre of Excellence, Brno, Czech Republic
Department of Computer Graphics and Multimedia, Faculty of Information Technology, Brno University of Technology, IT4Innovations Centre of Excellence, Brno, Czech Republic

 

Abstract: This article discusses problems and decisions related to scheduling of stream processing applications in heterogeneous clusters. An overview of the current state of the art of the stream processing on heterogeneous clusters with a focus on resource allocation and scheduling is presented first. Then, common scheduling approaches of various stream processing frameworks are discussed and their limited applicability in the heterogeneous environment is demonstrated on a simple stream application. Finally, the article presents a novel heterogeneity-aware scheduler for the stream processing frameworks based on design-time knowledge as well as benchmarking techniques. It is shown that the scheduler overcomes alternatives in resource-aware deployment over cluster nodes and thus it leads to a better utilisation of the clusters.

 

Keywords: scheduling; resource awareness; benchmarking; stream processing; Apache Storm; heterogeneous clusters; heterogeneity awareness; resource allocation.

 

DOI: 10.1504/IJBDI.2015.069090

 

Int. J. of Big Data Intelligence, 2015 Vol.2, No.2, pp.70 - 80

 

Submission date: 30 Sep 2014
Date of acceptance: 04 Dec 2014
Available online: 26 Apr 2015

 

 

Editors Full text accessAccess for SubscribersPurchase this articleComment on this article