Title: Prospects, challenges and latest developments in designing a scalable big data stream computing system
Authors: Dawei Sun; Chunxiao Liu; Dongfeng Ren
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China; School of Information Engineering, China University of Geosciences, Beijing 100083, China
College of Information Science and Technology, Bohai University, Jinzhou, Liaoning 121013, China
Network Security Department, Neusoft Corporation, Shenyang 110179, China
Abstract: In the big data era, big data stream computing, as a computing paradigm, is gaining traction for real-time and online data computing applications, and is specially designed to solve the dilemma of real-time data stream computing by processing data online within real-time constraints. It is used to compute large amounts of data in the form of continuous data streams. Each computation is represented by a data stream graph, usually a directed graph. In this paper, the computing paradigm of big data stream computation in data stream graph is given, and some application scenarios and big data stream characteristics are presented. A series of challenges in designing a scalable big data stream computing system in big data environments are summarised by referring to some research results of big data stream computing system, which include stateless system architecture, elastically adaptive scheduling strategy and fine-grained fault tolerance strategy. All these challenges will greatly help us to understand big data steam computing, and provide a high-level guidance in designing an excellent and useful big data stream computing system.
Keywords: big data stream computing; task scheduling; system design; stateless system architecture; adaptive scheduling; fine-grained fault tolerance.
Int. J. of Wireless and Mobile Computing, 2015 Vol.9, No.2, pp.155 - 160
Submission date: 13 Apr 2015
Date of acceptance: 06 May 2015
Available online: 19 Oct 2015