Analysis and improvement of methods for detecting shellcodes in computer systems

The article discusses the problem of detecting and filtering shellcode – malicious executable code that contributes to the emergence of vulnerabilities in the operation of software applications with memory. The main such vulnerabilities are stack overflow, database overflow, and some other operating system service procedures. Currently, there are several dozen shellcode detection systems using both static and dynamic program analysis. Monitoring of existing systems has shown that methods with low computational complexity are characterized by a large percentage of false positives. Moreover, methods with a low percentage of false alarms are characterized by increased computational complexity. However, none of the currently existing solutions is able to detect all existing classes of shellcodes. This makes existing shellcode detection systems weakly applicable to real network links. Thus, the article discusses the problem of analyzing shellcode detection systems that provide complete detection of existing classes of shellcodes and are characterized by acceptable computational complexity and a small number of false alarms. This article introduces shellcode classifications and a comprehensive method of detecting them based on code emulation. This approach expands the detection range of shellcode classes that can be detected by concurrently evaluating several heuristics that correspond to low-level CPU operations during execution of various shellcode classes. The presented method allows efficient detection of simple and metamorphic shellcode. This is achieved regardless of the use of self-modifying code or dynamic code generation on which existing emulation-based polymorphic shellcode detectors are based.