Title: Log anomaly detection and diagnosis method based on deep learning

Authors: Zhiwei Liu; Xiaoyu Li; Dejun Mu

Addresses: Network Security College of Northwestern Polytechnical University, Xi'an, Shaanxi, 710072, China; Information Construction and Management Division, Northwestern Polytechnical University, Xi'an, Shaanxi, 710072, China ' Network Security College of Northwestern Polytechnical University, Xi'an, Shaanxi, 710072, China ' Network Security College of Northwestern Polytechnical University, Xi'an, Shaanxi, 710072, China; Research and Development Institute of Northwestern Polytechnical University in Shenzhen, Shenzhen, Guangdong, 518057, China

Abstract: In order to improve the accuracy of log anomaly detection and diagnostic effectiveness, this paper proposes a deep learning-based log anomaly detection and diagnosis method. Firstly, analyse the log data and obtain the corresponding relationship between the log keys and log parameters. Secondly, using deep learning to capture association features, a convolutional neural network bidirectional long short-term memory (CNN-BiLSTM) deep learning model is constructed. Finally, learning context sequence feature information from both positive and negative directions through bidirectional input, and implementing log anomaly detection and diagnosis based on the results of context sequence feature information. The experimental results show that the accuracy of log anomaly detection in this method can reach 98.6%, the time required for log anomaly detection can reach 1.1 s, and the recall rate for log anomaly detection is 96.8%. The log anomaly detection effect is good.

Keywords: deep learning; one hot encoding; context sequence features; log exception.

DOI: 10.1504/IJDMB.2025.142978

International Journal of Data Mining and Bioinformatics, 2025 Vol.29 No.1/2, pp.119 - 132

Received: 07 Aug 2023
Accepted: 08 Nov 2023
Published online: 02 Dec 2024 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article