Title: An agricultural data storage mechanism based on HBase

Authors: Changyun Li; Qingqing Zhang; Pinjie He; Zhibing Wang; Li Chen

Addresses: College of Computer and Science, Hunan University of Technology, Zhuzhou 412007, China; Intelligent Information Perception and Processing Technology Hunan Province Key Laboratory, Zhuzhou 412007, China; Intelligent Perception and Networked Control, Hunan Province Key Laboratory of College, Zhuzhou 412007, China ' College of Computer and Science, Hunan University of Technology, Zhuzhou 412007, China; Intelligent Information Perception and Processing Technology Hunan Province Key Laboratory, Zhuzhou 412007, China; Intelligent Perception and Networked Control, Hunan Province Key Laboratory of College, Zhuzhou 412007, China ' College of Computer and Science, Hunan University of Technology, Zhuzhou 412007, China; Intelligent Information Perception and Processing Technology Hunan Province Key Laboratory, Zhuzhou 412007, China; Intelligent Perception and Networked Control, Hunan Province Key Laboratory of College, Zhuzhou 412007, China ' College of Computer and Science, Hunan University of Technology, Zhuzhou 412007, China; Intelligent Information Perception and Processing Technology Hunan Province Key Laboratory, Zhuzhou 412007, China; Intelligent Perception and Networked Control, Hunan Province Key Laboratory of College, Zhuzhou 412007, China ' College of Computer and Science, Hunan University of Technology, Zhuzhou 412007, China; Intelligent Information Perception and Processing Technology Hunan Province Key Laboratory, Zhuzhou 412007, China; Intelligent Perception and Networked Control, Hunan Province Key Laboratory of College, Zhuzhou 412007, China

Abstract: With the development of agricultural space-time localisation, sensor network and cloud computing, the amount of agricultural data is increasing rapidly and the data structure becomes more complicated and changeable. Currently, the widely used agricultural database is the relational database. This database handles a large amount of data with very limited throughput and is not suitable for organisation and management of distributed data. HBase is a non-relational database of distributed file storage built on Hadoop platform. HBase is suitable for unstructured data storage database and it can handle large volume of data with high scalability. To better store agricultural big data in HBase, we propose a special agricultural data buffer structure which stores data based on the data category and a two-level indexing memory organisation strategy on HBase. The method proposed saves more than a quarter of the time compared to traditional buffering methods. Experimental results show the higher efficiency of the agricultural data buffer structure and memory-organisation strategy.

Keywords: agricultural big data; data buffer structure; HBase; two-level indexing strategy.

DOI: 10.1504/IJICT.2019.101864

International Journal of Information and Communication Technology, 2019 Vol.14 No.4, pp.456 - 469

Received: 26 Sep 2017
Accepted: 01 Nov 2017
Published online: 29 Aug 2019 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article