Authors: Mohammadreza Ramezani-Chemazi; Maede Ashouri-Talouki
Addresses: Department of IT Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran ' Department of IT Engineering, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran
Abstract: Shellcode is a code injected by the attackers to vulnerable software to gain access to the command prompt. The byte patterns of shellcodes help the intrusion detection systems to detect this type of shellcodes. To avoid detection, encoding algorithms is used by the attacker to encode the byte patterns. The detection of these encoded shellcodes is a challenging problem. To detect these encoded shellcodes, we perform a static analysis of encoding algorithms of Metasploit engine to extract the byte patterns (signature) of these algorithms. Then, we introduce a regular expression-based language called GtS to express these signatures. The experimental results show the effectiveness of our signatures in terms of accuracy and false positive rate.
Keywords: shellcode; Metasploit; encoding algorithms; static analysis; signatures.
International Journal of Security and Networks, 2018 Vol.13 No.2, pp.71 - 83
Accepted: 05 Jan 2018
Published online: 07 Jun 2018 *