Penetration Testing and Attack Automation Simulation: Deep Reinforcement Learning Approach

In this research, we propose a revolutionary deep reinforcement learning-based methodology for automated penetration testing. The suggested method uses a deep Q-learning network to develop attack sequences that effectively exploit weaknesses in a target system. The method is tested in a virtual environment, and the findings indicate that it can identify vulnerabilities that manual penetration testing is unable to. A variety of tools, including Deep Q-learning network, MulVAL, Nmap, VirtualBox, Docker, National Vulnerability Database (NVD), and Common Vulnerability Scoring System (CVSS), are used in this work. The suggested method significantly outperforms current automated penetration testing methods. Our proposed methodology can detect flaws that manual penetration testing misses and can be modified (in terms of penalty values) to adapt to the updates of the target system (network) changes. Additionally, it has the potential to greatly enhance penetration testing's effectiveness and efficiency and could contribute to the increased security of computer systems. Experimental tests conducted in this work reveal the effectiveness of DQN automated penetration testing by utilizing the most effective attack vectors in the attack automation process