Title: Research and optimisation of low-cost cloud log storage system based on data pattern awareness
Authors: Qixia Wang; Ning Lv; Junjun Du
Addresses: Shanxi Information Industry Technology Research Institute Co., Ltd., Shanxi, Taiyuan, China ' Shanxi Information Industry Technology Research Institute Co., Ltd., Shanxi, Taiyuan, China ' Shanxi Information Industry Technology Research Institute Co., Ltd., Shanxi, Taiyuan, China
Abstract: Logging services in public clouds have significantly enhanced the capabilities of research, operations, management and security assurance. However, cloud logs are characterised by large data size, long retention time, fast writing, low useful information density and high access latency. In order to reduce storage costs, the following requirements need to be met: 1) highly compressed save data to save space; 2) realise high-speed data writing and fast compression; 3) realise low-latency fast retrieval of compressed data. Realising these three requirements at the same time is challenging and needs to be customised according to specific application scenarios. In this study, we summarise the typical data patterns in cloud logs and propose a low-cost cloud log storage system based on data pattern awareness to meet the goals of high-compression rate, fast compression and low-latency retrieval. We comparatively test various low-cost cloud log storage systems in terms of compression rate, compression speed and retrieval latency and ultimately present experiences and outlooks for future research in related areas.
Keywords: cloud log; data pattern; log storage; high-compression ratio; low-query latency.
DOI: 10.1504/IJWMC.2024.141455
International Journal of Wireless and Mobile Computing, 2024 Vol.27 No.3, pp.229 - 243
Received: 31 Oct 2023
Accepted: 27 Jan 2024
Published online: 13 Sep 2024 *