Title: A HDFS dynamic load balancing strategy using improved niche PSO algorithm in cloud storage

Authors: Zhiyu Jian; Yiwei Jian

Addresses: Graphic and text information Center, Ordos Vocational College of Eco-Environment, Inner Mongolia, 017010, China ' College of Computer Science and Technology, Inner Mongolia Normal University, Inner Mongolia, Hohhot, 010000, China

Abstract: A Hadoop distributed file system (HDFS) NameNode dynamic load balancing strategy (NDLBT) using improved niche particle swarm optimisation (PSO) is proposed, which is used to solve the load balancing problem of urban surveillance video big data. Firstly, the relationship between bandwidth consumption and the bit rate of video files, the size of data blocks and access hotspots of online video files are analysed. Then, adaptive backup is realised by dynamic multiple copies of heterogeneous nodes. The dynamic distribution of metadata is implemented to ensure the performance of metadata server clusters. Finally, we propose a new improved niche PSO algorithm to achieve load balancing scheduling. In the experimental scenario where high bandwidth consumption video files are used as service access hotspots, the proposed method, which can reduce the bandwidth peak value of bottleneck nodes in data node clusters by 20%, is superior to the original load balancing method in 90% scenarios.

Keywords: cloud storage; urban surveillance video; improved niche; PSO; particle swarm optimisation; dynamic load balancing; big data; HDFS; Hadoop distributed file system.

DOI: 10.1504/IJAACS.2021.114291

International Journal of Autonomous and Adaptive Communications Systems, 2021 Vol.14 No.1/2, pp.163 - 178

Received: 09 Mar 2020
Accepted: 03 Jun 2020

Published online: 15 Apr 2021 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article