Article: Optimisation of distributed storage technology for large-scale data based on Hadoop technology Journal: International Journal of Reasoning-based Intelligent Systems (IJRIS) 2025 Vol.17 No.7 pp.11 - 20 Abstract: The fast growth of big data technology makes effective storage and processing of vast amounts of data a major concern in contemporary computing systems. The growing data scale results in specific bottlenecks in data storage, task scheduling, and resource allocation even in the conventional distributed systems. Thus, this work presents a large-scale data distributed storage optimisation model based on Hadoop, i.e., Hadoop-OptiStor, which intends to improve the efficiency of Hadoop in big data processing by optimising important technologies such data distribution and store copy management. By means of experimental verification, the Hadoop-OptiStor model presented in this work exhibits notable improvement in numerous important criteria and performs better than the conventional distributed system. The optimisation model has higher application possibilities and practical value, according to the testing results; it can also efficiently increase resource utilisation, lower compute and storage bottlenecks. Inderscience Publishers - linking academia, business and industry through research

Title: Optimisation of distributed storage technology for large-scale data based on Hadoop technology

Authors: Yanke Qi

Addresses: School of Computer Science, Zhengzhou University of Aeronautics, Zhengzhou, 450046, China

Abstract: The fast growth of big data technology makes effective storage and processing of vast amounts of data a major concern in contemporary computing systems. The growing data scale results in specific bottlenecks in data storage, task scheduling, and resource allocation even in the conventional distributed systems. Thus, this work presents a large-scale data distributed storage optimisation model based on Hadoop, i.e., Hadoop-OptiStor, which intends to improve the efficiency of Hadoop in big data processing by optimising important technologies such data distribution and store copy management. By means of experimental verification, the Hadoop-OptiStor model presented in this work exhibits notable improvement in numerous important criteria and performs better than the conventional distributed system. The optimisation model has higher application possibilities and practical value, according to the testing results; it can also efficiently increase resource utilisation, lower compute and storage bottlenecks.

Keywords: Hadoop; large-scale data; distributed storage.

DOI: 10.1504/IJRIS.2025.146672

International Journal of Reasoning-based Intelligent Systems, 2025 Vol.17 No.7, pp.11 - 20

Received: 30 Mar 2025
Accepted: 25 Apr 2025
Published online: 11 Jun 2025 *

Title: Optimisation of distributed storage technology for large-scale data based on Hadoop technology

Keep up-to-date