AraCTI-NER: A Dataset and Benchmark for Arabic Cyber Threat Intelligence Named Entity Recognition

Abstract

Automated extraction of structured threat information from unstructured cyber threat intelligence (CTI) underpins modern security operations, yet the supporting machine learning resources are almost exclusively English: no annotated Arabic CTI named entity recognition (NER) corpus has been published. We introduce AraCTI-NER, a dataset of 10,312 token-level annotated samples (275,530 tokens; 42,360 entity spans) over eight STIX-inspired entity types, built by an LLM-assisted pipeline seeded with authentic Arabic cybersecurity articles, structurally validated and rebalanced through targeted generation. We benchmark seven encoders from three families (Arabic-specialized, English cybersecurity-adapted, and multilingual) over three seeds under strict entity-level metrics, and release a 408-sentence expert-audited test subset (ATS-gold) whose reliability is quantified by a second independent expert validation (inter-annotator agreement 0.878 entity-level F1). XLM-RoBERTa Large attains the best mean F1 (0.7603; 0.7674 on ATS-gold), with AraBERTv2 close behind (0.7491), while both English-only cybersecurity encoders fall to ≈0.63, a separation that holds across every seed and survives expert correction, with the ≈3-point F1 decrease from ATS-silver to ATS-gold concentrated in Vulnerability and TTP. On 350 doubly annotated sentences from authentic Arabic cyber-incident news, a shift in both provenance and register, the strongest model reaches F1 = 0.5429 against an inter-annotator F1 of 0.616. AraCTI-NER establishes the first reproducible baseline for Arabic CTI NER and identifies domain-adaptive Arabic cybersecurity pre-training as the highest-value next step.