Title: A distributed data processing platform over meteorological big data using MapReduce

Authors: Tao Huang; Shengjun Xue; Xiang Li; Feng Luo

Addresses: School of Computer Science and Technology, Silicon Lake College, Suzhou, China ' School of Computer Science and Technology, Silicon Lake College, Suzhou, China ' School of Computer and Software, Nanjing University of Information Science and Technology, Nanjing, China ' Meteorological Station, Shanghai Jiading Meteorological Service, Intersection of Shisheng Road and Shengzhu Road, Jiading 201800, China

Abstract: In the era of big data, the data of the meteorological departments grows explosively, which puts higher requirements on the real-time processing of meteorological big data. Besides, the efficient storage for the massive meteorological data has also attracted much attention from the meteorological departments. Therefore, in response to the urgent requirements of meteorological big data in processing and storage, a distributed data processing platform over meteorological big data using MapReduce is designed. Technically, the platform develops corresponding real-time strategies according to various data properties, and obtains meteorological big data in real-time from multiple channels. Based on the MapReduce real-time processing, we realise the distributed storage to store the meteorological big data in the platform. Overall, our platform improves the real-time, reliability, availability and the access efficiency of meteorological big data which is easy to expand and also has a good reference value for big data processing in other similar industries.

Keywords: meteorology; massive; big data; data processing; cloud; platform; distributed; MapReduce; real-time; sharing.

DOI: 10.1504/IJIITC.2019.104720

International Journal of Intelligent Internet of Things Computing, 2019 Vol.1 No.1, pp.74 - 85

Received: 23 Jan 2019
Accepted: 24 Feb 2019
Published online: 29 Jan 2020 *