Title: Research and optimisation of low-cost cloud log storage system based on data pattern awareness

Authors: Qixia Wang; Ning Lv; Junjun Du

Addresses: Shanxi Information Industry Technology Research Institute Co., Ltd., Shanxi, Taiyuan, China ' Shanxi Information Industry Technology Research Institute Co., Ltd., Shanxi, Taiyuan, China ' Shanxi Information Industry Technology Research Institute Co., Ltd., Shanxi, Taiyuan, China

Abstract: Logging services in public clouds have significantly enhanced the capabilities of research, operations, management and security assurance. However, cloud logs are characterised by large data size, long retention time, fast writing, low useful information density and high access latency. In order to reduce storage costs, the following requirements need to be met: 1) highly compressed save data to save space; 2) realise high-speed data writing and fast compression; 3) realise low-latency fast retrieval of compressed data. Realising these three requirements at the same time is challenging and needs to be customised according to specific application scenarios. In this study, we summarise the typical data patterns in cloud logs and propose a low-cost cloud log storage system based on data pattern awareness to meet the goals of high-compression rate, fast compression and low-latency retrieval. We comparatively test various low-cost cloud log storage systems in terms of compression rate, compression speed and retrieval latency and ultimately present experiences and outlooks for future research in related areas.

Keywords: cloud log; data pattern; log storage; high-compression ratio; low-query latency.

DOI: 10.1504/IJWMC.2024.141455

International Journal of Wireless and Mobile Computing, 2024 Vol.27 No.3, pp.229 - 243

Received: 31 Oct 2023
Accepted: 27 Jan 2024

Published online: 13 Sep 2024 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article