STINER, a taxonomy and expert-annotated corpus for extracting strategic intelligence from social media streams is introduced, and how social-media-driven extraction can surface early signals of the SafePay ransomware campaign prior to its retrospective characterization in vendor threat landscape reports is illustrated.
Abstract
Strategic Cyber Threat Intelligence (CTI) focuses on high-level insights, such as identifying targeted industries, attributing attacks to specific ransomware groups, and assessing the scale of data loss. Today, X (formerly Twitter) has become the fastest source for this intelligence, often hosting real-time breach announcements days before formal vendor reports. Converting this raw chatter into actionable intelligence requires navigating a complex linguistic landscape. Conventional Named Entity Recognition (NER) models struggle to parse the informal and highly irregular dialect of social media, creating a blind spot for automated defense systems. To address this challenge, we introduce STINER, a taxonomy and expert-annotated corpus for extracting strategic intelligence from social media streams. We construct a high-quality, expert-annotated dataset of 2,100 real-world alerts and propose a granular taxonomy of eight entity types centered on strategic pivots such as Threat Actor, Sector, and Location. We benchmark nine models across 12 evaluated configurations, spanning general-purpose and domain-adapted encoders, open-schema extraction, and generative LLMs in both zero-shot and fine-tuned settings. Domain-adapted encoders such as DarkBERT reach a strict F1-score of 89.33%, outperforming both general-purpose baselines and fine-tuned Large Language Models, which additionally incur substantially higher inference latency. Leveraging STINER-DarkBERT, we conduct a European threat landscape analysis for H1 2025. Our results align with official reporting on major targets while highlighting the distinct visibility profile of attacks in Spain, and illustrate how social-media-driven extraction can surface early signals of the SafePay ransomware campaign prior to its retrospective characterization in vendor threat landscape reports.
APT reports in the Cyber Threat Intelligence (CTI) are rich with information, but are not structured, resulting in security analysts devoting 2-3 hours to manually sifting through each report to identify the linkages, tactics, techniques, and relationships. Named Entity Recognition (NER) systems aim to overcome the manual nature of this task, but existing solutions result in limited entity coverage, boundary false negatives caused by the rigid fixed-window tokenization, and failure to resolve conflicting vendor aliases. Therefore, this paper introduces CyberNER, a two-stage pipeline to solve the multi-type NER and alias canonicalization problem in APT CTI reports. In Stage 1, Documents through semantic hierarchical chunking are categorised at natural section, paragraph, and sentence boundaries. Stage 2 would introduce fine-tuned CySecBERT, a cybersecurity language model based on CySecBERT, to identify 12 types. Each entity has a confidence score. There is also a three-tier MITRE STIX 2.1 alias canonicalization layer that addresses variations in surface forms from different vendor reports. The Threat Recall Actor improves exponentially from a baseline of 21% to 94%. Evaluated using a 12-type annotated benchmark from 182 APTNotes reports covering 30 APT groups, CyberNER achieves a Macro-F1 score of 0.853, outperforming all four baselines. The structured JSON output is directly consumable for downstream tasks, including TTP mapping, attack-flow graph generation, and multi-source intelligence fusion.
Unnamalai K, Suriakala M· International journal of com...· 0 citations
Cyber Threat Intelligence (CTI) enables Security Operations Centers (SOCs) to understand adversary behavior,
prioritize risks, and respond to cyber threats. However, cur-rent CTI workflows still depend heavily on manual analysis
of unstructured threat reports, vulnerability advisories, open-source intelligence, social media posts, and structured feeds. This
creates operational latency, inconsistent extraction quality, weak provenance, and limited scalability. This paper presents Auto
Threat AI, an agentic and explainable framework for automated CTI extraction, correlation, scoring, and analyst-governed SOC
operationalization. The proposed framework integrates determin-istic indicator extraction, Natural Language Processing (NLP),
schema-guided Large Language Model (LLM) agents, graph-aware threat correlation, bounded risk scoring, evidence-first
explainability, and Human-in-the-Loop (HITL) governance. The system ingests heterogeneous CTI sources, extracts entities and
relations such as IOCs, CVEs, malware, campaigns, threat actors, tools, and techniques, constructs a threat knowledge graph,
gen-erates campaign candidates, and presents risk-ranked intelligence through a SOC dashboard. Experimental evaluation on
safe demonstration CTI data shows that the implemented prototype ingested 6 sources, extracted 36 entities, generated 33
relations, detected 11 threat events, identified 5 campaign candidates, and routed 8 items for HITL review. The results
demonstrate that Auto Threat AI can reduce manual CTI processing effort while improving traceability, explainability, and
analyst trust.
Methari Keeravani· International Journal for Re...· 0 citations
Cyber Threat Intelligence reports combine analytical prose with dense technical indicators, making structured entity extraction a challenging but operationally valuable task. This study presents a comparative evaluation of three large language models – Claude Sonnet 4.6, GPT-5.4, and LLaMA 4 Scout – on a manu-ally annotated corpus of 21 real-world CTI reports across 15 entity types and 1284 ground truth instances. This study evalu-ates zero-shot and few-shot prompting conditions and studies the effect of iterative prompt refinement, focusing on explicit format constraints for cryptographic hash entities. Results show that Claude Sonnet 4.6 and GPT-5.4 achieve comparable perfor-mance under zero-shot conditions, with LLaMA 4 Scout trailing by a substantial margin. Few-shot prompting consistently reduc-es hallucination rates, but yields mixed F1 results, with exemplar cardinality emerging as a critical and underappreciated design factor. Entity extraction difficulty varies substantially across types, with technical indicator categories showing near-perfect performance and semantic categories such as tool and target sector posing the greatest challenges across all evaluated mod-els.
Aykhan Huseynli· International Journal of Adv...· 0 citations
Cyber threat intelligence (CTI) reports are typically written in unstructured formats, which complicates the extraction and analysis of important entities and adversarial behaviors. Although existing CTI research provides extraction tools, knowledge-graph frameworks, and MITRE ATT&CK mapped datasets, curated report-level datasets that preserve complex entity relationships and normalized adversarial behaviors remain limited. To address this limitation, this study presents a manually constructed dataset of 150 English-language CTI reports, each represented as STIX 2.1 based graphs, which includes 4,777 STIX entities, 5,817 STIX relationships in total, and 1,273 STIX attack-pattern entities (adversarial behaviors) mapped to 269 unique MITRE ATT&CK Enterprise techniques and sub-techniques. Twenty five randomly sampled reports were independently assessed by two cybersecurity researchers, which shows substantial inter-rater agreement. Disagreements were subsequently adjudicated to establish a gold-standard reference dataset. Four locally deployed open-source LLMs were evaluated as automated judges against this adjudicated reference sample. Qwen3.6:27B achieved the strongest overall performance, with a maximum kappa score of 0.803, micro-F1 scores exceeding 92%, and false-positive rates below 5%. The dataset provides a benchmark for CTI information extraction, knowledge-graph construction, incident analysis, and threat attribution. The findings further indicate that locally deployed LLMs can support human reviewers in identifying annotation inconsistencies, but expert validation remains essential.
D. Das, Arnab Banik, Md. Shariful Islam et al.· 0 citations
Cyber threat intelligence from underground forums has traditionally relied on passive monitoring. However, as users have become more aware of large-scale data collection, valuable intelligence has become increasingly rare in open forums, often migrating instead to private or harder-to-reach spaces, making passive approaches inadequate. Building on the intuition that relevant information can be obtained through active elicitation, this paper presents DarkBot, to the best of our knowledge, the first multi-agent LLM-based system for active CTI elicitation in underground forums. DarkBot decomposes the interaction task across eleven specialized agents organized into three functional blocks: engagement gating for relevance and safety filtering, context-aware question generation driven by MITRE ATT&CK tactics, and linguistic style adaptation to better align with real forum users. In a controlled evaluation across 100 CrimeBB conversations, the system recovered 72.8% of the validated MITRE ATT&CK techniques present in the original discussions by observing only the initial post at the start of each interaction, and it consistently outperformed a monolithic baseline. The proposed layered safety design contained all injected jailbreak attempts at the pipeline level. These results were further supported by real-world experiments: in a prospective matched deployment, threads assigned to DarkBot accumulated an average of 3.85 more CTI entities than their controls over seven days, and across 104 live forum conversations, the system elicited CTI-relevant disclosures without observed account suspensions, moderator interventions, or explicit accusations of automated participation.
Lorenzo Bossi, Federico Saccani, Francesco Panebianco et al.· 0 citations
This paper introduces a Customer Threat Detection Model leveraging a pre-trained large language model (LLM) on a major cloud platform to analyze customer service call transcripts and social media posts for potential security threats. The solution was developed in response to a critical need by the corporate security team to proactively identify threats during high-risk periods—such as the Southern California wildfires in January—when call volumes to the Customer Contact Center surged and employees and property faced elevated safety risks. Historically, manual identification of threats was slow and inconsistent, creating potential exposure for the organization. Operating in batch mode, the system processes daily calls and assigns each interaction a threat score (0–100), mapped to five ordinal bins from Low to High. The model combines expert-defined keywords with semantic embedding techniques to expand its threat lexicon, enabling detection of evolving language and context. Each transcript is transformed into a structured prompt and evaluated by the LLM to produce a threat score and category.Manual review sampled calls showed ~93% accuracy but proved resource-intensive and impractical for ongoing monitoring. To address scalability, we applied an “LLM-as-a-Judge” framework, where LLMs act as surrogate evaluators of model outputs. For 10K sampled calls, two summaries per call, overall and threat-focused, were generated and independently assessed by a second LLM to assign ordinal threat categories. Agreement metrics (accuracy, Cohen’s kappa, mean absolute difference), triadic consistency, and keyword sensitivity were computed. A small Keyword Influence Delta indicated strong contextual detection and guided keyword refinement.Results indicate good agreement between the deployed model and independent LLM judges, demonstrating scalability and reduced analyst workload in safety‑critical monitoring contexts.