By Gábor Pék, CrySyS Lab, Budapest University of Technology and Economics
Automated penetration testing is one of the recent challenges of cyber-security with the visionary goal of discovering and exploiting system- and software vulnerabilities with no human involvement at all. Over the last couple of years various automated and semi-automated (e.g., Nessus, Acunetix Appscanner) tools and services were suggested by specialized vendors each of which focused on a slightly different aspect of possible security exposures. Though, the concept is promising and seemingly working, most of these solutions typically brought extra workload for human testers forcing them to doublecheck the automatically generated results. An early, undoubtly acknowledgeable milestone in this field was demonstrated by an Artificial Intelligence (AI) engine called Mayhem at the 2016 DEFCON CTF Finals [1] which could nicely keep up with 14 other human teams in the world championship of hacking in Las Vegas. Recently, however, by realizing and riding the stunning opportunities of Natural Language Processing and Large Language Models (LLMs) the dawn of a new AI era has sprouted forth. Without overwhelmingly repeating and highlighting what the reader might already know, AI is a double-edged sword that can bring various pros and cons depending on the intent of its use. Some eye-opening results have already been demonstrated nowadays where teams of LLM agents successfully exploited One- and Zero-day vulnerabilities [3, 4] using top-tier AI models. Others, created semi-automatic AI tools such as PentestGPT [2] to help humans in their pentesting endeavor. An automated tool called AutoAttacker [5] uses sub-agents and situation-aware reasoning to render AI prompts to perform complex cyber-attacks such as privilege escalation, credential stealing and ransomware attacks.
Our goal now is to support IoT systems by automating their security testing in their design phase. To do this, we first build a multi-tier model from the IoT system comprising:
i.) Functional and information flow model that describes the high-level functions and flow of information as well as produces/consumes relationships between these functions and information.
ii.) System model that includes concepts such as Hosts, Networks, Agents (e.g., humans, processes), Services and Data. Services are the implementation of high-level functions relating to the functional model, and Data represents the information as expressed by the information flow model. With these concepts, one can describe the static configuration of the system (e.g., host:H is_connected_to network:N or process:P provides service:S), as well as its dynamic features, where the latter can be represented by rules (e.g., process:P can_use service:S if process:P can_access service:S and process:P can_present(credential:C)).
iii.) Deployment/implementation model to handle devices and actual software versions (e.g., Raspberry Pi computer, SQLite database version X) and to implement relationships between devices and host, as well as software and processes.
We collect all the information required from the IoT system operators in advance both to build this aforementioned multi-tier model as well as to generate and spin-up the digital twin of the original IoT system. At this point, we render safety-critical goals using impact analysis that breach the predefined system requirements of operators (e.g., sensor data can be collected). Later, we translate these goals into the domain of cyber-security to see whether they can be actualized by a malicious actor. Then, starting from one such goal, we build an attack tree by adding preconditions as nodes using the rules and expressions of the system model as Figure 1 shows it. Once the tree cannot be refined further in a declarative way (e.g., in Prolog), we bend upon collaborative AI bots that generate more concrete attack steps from the deployment model and extend the attack tree accordingly.
Figure 1 – Building combined attack trees. The original attack goal rendered by impact analysis comprises the root of the tree. The corresponding preconditions for such an attack goal are calculated in a declarative way based on the expressions and rules of the system model. Once a goal cannot be refined further in the domain of the system model, we start generating attack steps using collaborative AI models.
To achieve this, we assign different responsibilities for different AI bots as shown in Figure 2 and instruct them from a manually programmed orchestration logic to complete their task. This way we try to minimize the hallucination of these LLMs not to derail from their original task. Our multi-bot approach enables us to select the best performing LLM for each specific task. Currently, we chose the LLMs provided by Open AI (GPT-4o) and Anthropic (Claude 3.5 Sonnet) due to their high accuracy and performance. For example, the bot called ”A-T summarizer” dissects the input IaC files (i.e., Terraform, Ansible) of system operators in order to retrieve low-level details of the infrastructure so we can enrich the LLMs with context information (e.g., IP addresses of hosts, network configuration, software versions and so on). Furthermore, as we continuously refine on the preconditions for a given system model goal (e.g., ATTACKER can comporomise credential:C), we iteratively custom-tailor the AI prompts using various Prompt Engineering techniques such as CoT (Chain of Thoughts). Here we apply another guardrail by instructing the corresponding AI bots to propose attack techniques only from the collection of the MITRE ATT&CK framework. The following example shows an intermediatory prompt enriched with context information (e.g., voter device) generated by the „Prompt creator” bot.
Figure 2 – Description of and interactions within our multi-bot architecture. Each AI bot comes with a specific task to complete.
If we reach an attack technique that cannot be refined further, we generate a fully customized and parameterized command based on the toolset of the Kali Linux distribution. This way all the leaves of the attack tree will point to a specific tool that can be later executed as attack scenarios against the digital twin of the IoT system.
Our vision is that all the steps we described above can be fully automated given that all the necessary input information describing the IoT system is at hand and correct. Also, by combining system modeling, the power of declarative languages, and the opportunities offered by LLMs, we expect more complete and accurate test results compared to previous tools and solutions.
References:
[1] https://www.darpa.mil/news-events/2016-08-04[2] Deng, Gelei, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024. “PentestGPT: An LLM-Empowered Automatic Penetration Testing Tool.” arXiv. http://arxiv.org/abs/2308.06782.
[3] Fang, Richard, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. “LLM Agents Can Autonomously Exploit One-Day Vulnerabilities.” arXiv. http://arxiv.org/abs/2404.08144.
[4] Fang, Richard, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. “Teams of LLM Agents Can Exploit Zero-Day Vulnerabilities.” arXiv. http://arxiv.org/abs/2406.01637.
[5] Xu, Jiacen, Jack W. Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, and Zhou Li. 2024. “AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-Attacks.” arXiv. http://arxiv.org/abs/2403.01038.
