Title: A configurable and executable model of Spark Streaming on Apache YARN

Authors: Jia-Chun Lin; Ming-Chang Lee; Ingrid Chieh Yu; Einar Broch Johnsen

Addresses: Department of Information Security and Communication Technology, Norwegian University of Science and Technology (NTNU), 2802 Gjøvik, Norway ' Department of Information Security and Communication Technology, Norwegian University of Science and Technology (NTNU), 2802 Gjøvik, Norway ' Department of Informatics, University of Oslo, 0315 Oslo, Norway ' Department of Informatics, University of Oslo, 0315 Oslo, Norway

Abstract: Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, finding these parameters happens by trial and error on the complex, deployed framework. This paper shows that high-level models can help to determine these parameters by predicting and comparing the performance of streaming applications running on stream processing frameworks with different configurations. To demonstrate this approach, this paper considers Spark Streaming, a widely used framework to leverage data streams on the fly and provide real-time stream processing. Technically, we develop a configurable and executable model to simulate both the streaming applications and the underlying Spark stream processing framework. Furthermore, we model the deployment of Spark Streaming on Apache YARN, which is a popular open-source distributed software framework for big data processing. We show that the developed model provides a satisfactory accuracy for predicting performance by means of empirical validation.

Keywords: modelling; simulation; Spark Streaming; Apache YARN; batch processing; stream processing; ABS.

DOI: 10.1504/IJGUC.2020.105531

International Journal of Grid and Utility Computing, 2020 Vol.11 No.2, pp.185 - 195

Received: 21 Jun 2018
Accepted: 06 Dec 2018
Published online: 03 Mar 2020 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article