Title: Malware analysis and detection using optimised dynamic path-controllable deep unfolding neural network in PE files using YARA rules

Authors: Vivek Kumar Anand; Sanjay Kumar Bishwas

Addresses: Department of Computer Science and Engineering, NIIT University, Neemrana-301705, India ' Department of Computer Science and Engineering, NIIT University, Neemrana-301705, India

Abstract: The rapid evolution of malware necessitates an optimised approach for effective detection. This study proposes malware analysis and detection using an optimised dynamic path-controllable deep unfolding neural network in PE files with YARA rules (DPCDUNN-MA-PEF). Initially, PE file data undergoes pre-processing using the generalised multi-kernel maximum correntropy Kalman filter (GMKCKL) to remove redundancy. Relevant features are extracted using the multi-objective matched synchrosqueezing chirplet transform (MOMSSCT). The extracted features are analysed using the dynamic path-controllable deep unfolding network (DPCDUN) for malware classification. To enhance detection accuracy, the hunger games search optimisation algorithm (HGSOA) optimises DPCDUN parameters. The proposed method is implemented in Python and examined using performance metrics such as accuracy, precision, recall, F1-score, error rate, ROC, computational time. Experimental results show superior performance, with up to 29.28% higher F1-score compared to YARA-FH-FRMA, DGL-IDA-MD, and ERMD-CFT-DNN.

Keywords: dynamic path-controllable deep unfolding network; generalised multi-kernel maximum correntropy Kalman filter; hunger games search optimisation; multi-objective matched synchrosqueezing chirplet transform.

DOI: 10.1504/IJAHUC.2025.149465

International Journal of Ad Hoc and Ubiquitous Computing, 2025 Vol.50 No.2, pp.91 - 102

Received: 19 Feb 2024
Accepted: 25 Feb 2025
Published online: 01 Nov 2025 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article