By Miltiadis Siavvas, Information Technologies Institute (ITI) of the Centre for Research and Technology-Hellas (CERTH)
In this insight post, we present the approach CERTH proposes for enabling the automation of the identification and extraction of security requirements from security standards with Large Language Models (LLMs).
Compliance with international security standards is crucial for modern information systems for enhancing their cybersecurity posture. But with the manual extracting of security requirements from lengthy and highly complex standards documents, the process of ensuring compliance remains a time consuming, effort-demanding and error-prone task. Although several techniques from the field of natural language processing (NLP) have been studied for enabling the extraction of requirements from textual descriptions, none of them achieved sufficient accuracy and automation. Recently, the proposition of the Transformer-based models, particularly Large Language Models (LLMs), which have demonstrated remarkable human-like capabilities in language understanding and processing, has created new opportunities for automating the process of digital transformation of security standards.
CERTH’S solution, an AI mechanism, automatically identifies and extracts the main requirements that are expressed in the standard. The idea is to adapt a Transformer-based LLM like BERT, GPT, and BART, and teach it how to detect and extract requirement sentences from the textual descriptions found in security standards.
The overall approach for building such an LLM-based solution is illustrated in Figure 1. As can be seen in Figure 1, the overall approach consists of four different steps, which are: (i) Dataset Construction, (ii) Dataset Preparation, (iii) Model Selection, and (iv) Model Execution (Inference). The first three steps are responsible for building the Requirements Identification and Extraction (RIE) model, whereas the last step corresponds to the actual utilization of the produced model in practice.
Figure 1: The high-level overview of the proposed approach
Step 1: Dataset Construction
The first step is the construction of a domain-specific dataset. Using two security standards, ISO/IEC 27001 and NIST SP 800 53, CERTH created a dataset of 500 pairs of actual paragraphs (context) from the standard and the security requirements (sentences) that those paragraphs contained. This domain-specific dataset is required for fine-tuning the selected LLMs.
Step 2: Dataset Preparation
This step is responsible for preparing the data for the model selection process. The dataset is split into training, validation, and testing sets, with the first two used for fine-tuning the LLMs and selecting the best model, whereas the third for evaluating the model’s performance on unseen data. CERTH also tokenized the data samples to generate word sequences, which is the required format for LLMs.
Step 3: Model Selection
Several popular pre-trained LLMs – BERT, DistilBERT, ALBERT, RoBERTA, T5, Falcon, and BART – were fine-tuned for the specific task of security requirements identification and extraction, based on the constructed domain-specific dataset.
For this goal, two different fine-tuning approaches were explored:
- Question-Answering (QA), and
- Summarization
Despite the relatively small dataset, the LLM models showed good performance, with BART demonstrating the best results.
Step 4: Model Execution (Inference)
To demonstrate the effectiveness of the proposed approach, CERTH applied the BART-based model to identify and extract security requirements from an unseen security standard —ETSI TS 103 701.
Figure 2 presents an example from a textual fragment retrieved from ETSI TS 103 701, containing two security requirements within the text (marked in red) and the output of the model.
Figure 2: Example of requirements extraction from a security standard
The model accurately extracted two distinct security requirements hidden within the paragraph, filtering out irrelevant text.
______________________________________________
This example shows that the application of LLMs and the highly effective RIE models may automate the labor-intensive process of manually extracting requirements from standards, thereby greatly streamlining the process, and reducing the time and effort required to perform it. The automatic extraction of security requirements from security standards is important for streamlining the security validation and certification process, as these requirements are utilized to construct the checklist on which the evaluation is based.
The results achieved so far are promising, despite the small dataset that has been used, and with the future extension of the dataset the accuracy of the LLMs could be further enhanced. During the DOSS project, a large requirements dataset will be constructed for building highly accurate and generalizable RIE models, which will be important project outcomes. In addition to this, larger models like Llama 3.0 and DeepSeek will be also investigated, along with additional LLM adaptation techniques (in conjunction with fine-tuning), with specific focus on zero-shot and few-shot learning.
This Insight post is based on the conference paper “Digital Transformation of Security Standards: Requirements Extraction using Large Language Models” written by Siavvas M, Xanthopoulou G, Kalouptsoglou I, Kehagias D, Tzovaras D., that was presented at the 11th International Conference on Dependable Systems and Their Applications (DSA2024), 2-3. November 2024, Taicang, Suzhou, China, and is available here.
