A common way by which attackers gain control of hosts is through remote exploits. In this paper we consider such exploits (shellcodes) which exploits a buffer overflow vulnerability. In the paper we identify and describe the common and specific shellcode features, and also we propose a shellcode classification which is based on those features. We propose a shellcode detection method that solves the problem of false positives rate minimization while providing the full coverage of detected shellcode classes. Proposed method solves the critical problem of algorithm’s running time minimization as well.
The proposed method has been implemented and tested on different data sets. Experiments results show that false positives rate of the proposed method is closed to 0 and efficiency in terms of time complexity is 16-45 times higher than linear combination of existing algorithms depending on the testing data.