Title: Analysing and evaluating topology structure of online application in Big Data stream computing environment

Authors: Rui Huang; Dawei Sun

Addresses: School of Information Engineering, China University of Geosciences, Beijing 100083, China ' School of Information Engineering, China University of Geosciences, Beijing 100083, China

Abstract: Stream computing systems process high-rate incoming data sources in dynamic environments and generate real-time results. A critical problem in stream processing is the difficulty in devising an optimal strategy for configuring topology structure. That is due to the inability to predict or adapt to the dynamics of the data flow intensity and the computing platform's workload. Therefore, the best strategy should rely on the real-time statistics of performance and workload to achieve self-adaptation. In this paper, we first analyse the impact of application topology structures in Storm on performance and resource utilisation efficiency. From the metrics of stream processing application and environment, we derive some effective optimisation rules towards designing an online reconfiguration strategy. Our evaluation of the experimental results reveals the intricacies in searching the configuration parameters space and shows that a strategy based on our rules can substantially improve performance as well as achieve high resource utilisation efficiency.

Keywords: big data; stream computing; parallelism adjustment; topology structure; Storm; online applications; data flow intensity; resource utilisation; optimisation rules; online reconfiguration strategy; efficiency; performance evaluation.

DOI: 10.1504/IJWMC.2016.078204

International Journal of Wireless and Mobile Computing, 2016 Vol.10 No.4, pp.317 - 324

Received: 16 Mar 2016
Accepted: 04 Apr 2016
Published online: 08 Aug 2016 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article