Authors: Ron Ziv; Oded Koren; Nir Perel
Addresses: School of Industrial Engineering and Management, Shenkar – Engineering. Design. Art., 12 Anne Frank St., Ramat-Gan, Israel ' School of Industrial Engineering and Management, Shenkar – Engineering. Design. Art., 12 Anne Frank St., Ramat-Gan, Israel ' School of Industrial Engineering and Management, Afeka Tel-Aviv Academic College of Engineering, 38 Mivtza Kadesh St., Tel-Aviv, Israel
Abstract: Handling data is becoming more and more complex. A higher velocity of data is created as more people have access to data generating devices such as computers, mobile phones, medical devices, home appliances, etc. Data files, such as user activity logs, system logs and so on, are stored in HDFS big data platform in various sizes, which takes into consideration the business requirements, infrastructure parameters, administration decisions, and other factors. Dividing the data files (in various volumes) without taking into consideration the HDFS™ predefined block size, may create performance issues that can affect the system's activity. This paper presents how HDFS™ block design affects the performance of Apache™ Hadoop® big data environment by testing different architectures for reading, writing, and querying identical datasets. We designed three scenarios to illustrate different file divisions on the big data platform. The findings present a significant impact on the performance of a system in accordance with the architecture deployed.
Keywords: information management; architecture; HDFS; performance; big data; block partition.
International Journal of Information Technology and Management, 2023 Vol.22 No.1/2, pp.140 - 159
Accepted: 14 May 2019
Published online: 05 Apr 2023 *