Skip to content
Open access

Open-set intrusion detection and semantic analysis of zero-day network attacks using deep reinforcement learning and large language models

TL;DR

A unified framework for open-set intrusion detection and semantic analysis of unknown network traffic by combining deep learning, reinforcement learning, and large language models is presented, which contributes to the development of adaptive, explainable, and deployable intrusion detection systems capable of addressing evolving cyber threats.

Abstract

The increasing prevalence of zero-day cyberattacks presents a significant challenge for modern Intrusion Detection Systems (IDS), as previously unseen network traffic often falls outside the distribution of supervised training data. Traditional deep learning-based IDS solutions operate under a closed-set assumption, requiring all attack categories to be known during training. Consequently, novel attacks may be misclassified as benign traffic or incorrectly assigned to known attack classes, reducing the effectiveness of network defense mechanisms. Although recent open-set recognition approaches improve the detection of unknown traffic, they typically provide limited insight into the behavioral characteristic of detected anomalies, requiring additional manual investigation by security analysts. This thesis presents a unified framework for open-set intrusion detection and semantic analysis of unknown network traffic by combining deep learning, reinforcement learning, and large language models. The proposed framework employs a Convolutional Neural Network (CNN) to learn feature representations from known network traffic classes and a Deep Q-Network (DQN) that uses SoftMax-derived uncertainty metrics, including maximum probability, probability gap, and Shannon entropy, to dynamically distinguish known from unknown traffic without manually defined confidence thresholds. To improve interpretability, a selective Large Language Model (LLM) reasoning module is applied only to traffic identified as unknown by the CNN-DQN pipeline. Unlike existing approaches that focus solely on detection or apply computationally expensive LLM reasoning across all traffic, the proposed framework combines efficient open-set detection with targeted semantic interpretation of suspicious network behavior. Experimental evaluation was conducted using the CICIDS-2017 and UNSW-NB15 intrusion detection datasets. The CNN-DQN framework achieved a binary F1-score of 97.83% for known-versus-unknown traffic separation while maintaining strong performance on known attack classes and effectively identifying previously unseen attacks. Cross-dataset experiments demonstrated the framework’s ability to generalize to traffic distributions not observed during training. The LLM-assisted reasoning stage generated behaviorally aligned explanations for 77.5% of DQN-flagged unknown traffic samples and provided meaningful behavioral insights for an additional 22.1% of samples, demonstrating its effectiveness in supporting contextual interpretation of suspicious network activity. The proposed framework contributes to the development of adaptive, explainable, and deployable intrusion detection systems capable of addressing evolving cyber threats. By combining uncertainty-aware open-set detection with semantic reasoning, the system enables security analysts, network administrators, and cybersecurity researchers to not only identify previously unseen attacks but also understand their underlying behavioral characteristics. This capability can support faster incident response, improved threat intelligence generation, and enhanced protection of enterprise, cloud, Internet of Things (IoT), and critical infrastructure environments against emerging cyber threats.

Read PDF

Similar papers

Open access Sep 2026

Explainable Sequence-Aware Deep Learning Framework for Potential Zero-Day Attack Detection and Cross-Domain Generalization in Enterprise Network Intrusion Detection

Zero-day attacks remain a significant challenge in enterprise network security because their previously unseen characteristics can reduce the effectiveness of conventional signature-based intrusion detection systems. Although machine learning and deep learning have improved intrusion detection, many existing approaches...

Uchechukwu Nwankwo, O. Nwokonkwo, C. Ikerionwu et al. · 0 citations
Open access 2026

Detecting Zero-Day Attacks via Reconstruction of Feature Influence and Model Uncertainty

In practical Network Intrusion Detection System (NIDS) deployments, detecting anomalies is only the first step, while determining the exact nature of those anomalies is equally important. Commonly, anomalous traffic is forwarded to a supervised multiclass classifier trained to identify known attack categories. While ef...

Hussein Fawaz, Jacopo Talpini, Marco Savi et al. · 0 citations
Conference Aug 2026

Deepsecure-IDS: A Deep Convolutional Learning Framework for High-Performance Network Intrusion Detection

Recent advancements in network technology have led to the emergence of numerous sophisticated intrusion types that differ significantly from existing threats. Traditional firewalls cannot immediately detect these intrusions with their own set of rules and policies. Therefore, an Intrusion Detection System (IDS) is esse...

R. Raja, S. Vairachilai · 0 citations
Review Open access Sep 2026

A Comprehensive Survey on Machine Learning and Deep Learning Approaches for Network Intrusion Detection Systems: Techniques, Datasets, and Open Challenges

Overall, NIDS research is gradually moving beyond the pursuit of higher accuracy toward practical challenges such as real-time edge inference, adversarial robustness, explainability, privacy-preserving and federated learning, and intrusion detection over encrypted traffic.

Y. Alnattaf, Hanaa Fathi Mahmoud · 1 citation
Open access Aug 2026

HADS-Net: A Hybrid Attention-Based Deep Security Network for Network Intrusion Detection

The principal contribution of this work is architectural and diagnostic rather than a performance improvement: it documents that combining feature-wise attention with out-of-fold stacked generalization does not, in this setting, outperform a plain multi-layer perceptron, while incurring the highest memory footprint of...

Mahima Khanna, V. Murthy, Siva Ramavarapu et al. · 0 citations
Open access 2026

A Framework for Simulated Zero-Day Detection Using Synthetic Attack Generation and Out-of-Distribution Evaluation

: Zero-day attacks pose a significant threat to computer systems and networks as they exploit weaknesses that have not been recognized by security professionals or software creators and for which there are no existing protective measures. This study introduced an innovative method for identifying Zero-day attacks throu...

Peter Kipngeno Langat, Michael W. Kimwele, D. Kaburu · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.