Title: Optimising the detection of metamorphic malwares using ensemble learning technique

Authors: Vinay Kumar; Abhishek Vaish

Addresses: Department of Information Technology, Indian Institute of Information Technology, Allahabad, India ' Department of Information Technology, Indian Institute of Information Technology, Allahabad, India

Abstract: Metamorphic malware is a significant challenge for traditional malware detection techniques, as it continuously changes its code to evade its detection. The behaviour-based approach involves analysing the behaviour of malware rather than its code. By monitoring the system's behaviour, it is possible to detect malicious activity that may be associated with malware. We have proposed API call-based technique to detect metamorphic malware. Our approach involves finding the top 30 malicious API calls having the highest probability score based on extra trees classifier and identifying patterns of malicious API calls that indicate malicious behaviour. This paper presents an API call-based detection technique and proposes a novel approach based on ensemble learning techniques. The proposed algorithm has an accuracy of 0.99 and the F1-score is 0.85. Our system can detect changes in the code structure and behaviour of the malware, even if the malware's binary code has been obfuscated into a new variant. We demonstrate the effectiveness using a benchmark dataset of metamorphic malware.

Keywords: metamorphic malware; gradient boosting; random forest; API calls.

DOI: 10.1504/IJICS.2025.146527

International Journal of Information and Computer Security, 2025 Vol.26 No.4, pp.316 - 331

Received: 30 May 2023
Accepted: 25 Aug 2023
Published online: 02 Jun 2025 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article