Skip to content
Open access

CTI-Rationale: An Expert-Annotated Dataset and Evaluation Framework for Evidence-Grounded MITRE ATT&CK Technique Mapping

Nov 2026 · Journal of Cybersecurity and Privacy · 0 citations · 31 references

Abstract

Mapping cyber-threat intelligence (CTI) to MITRE ATT&CK is usually evaluated as label prediction, yet a correct technique identifier may rest on the wrong words. We treat each mapping as an argument (a technique claim supported by textual data, a warrant, and rejected alternatives) and annotation as discovery followed by judgment. CTI-Rationale instantiates this model with 1740 expert paragraph–technique mappings from 100 public reports, each with an evidence span, free-text rationale, reasoning type, and confidence, and most with primitives and ruled-out alternatives. All rationale content was written and adjudicated by human analysts, not generated by artificial intelligence. The low paragraph-level Jaccard (0.421) is associated mainly with coverage differences, not conflicting labels: paragraphs mapped by both annotators reach 0.780, only 10.4% of one-sided mappings are conflicts, and capture–recapture estimates put gold completeness at 84.2–88.0% (likely optimistic), lowest for behavioral and contextual warrants. In cross-validation, the annotation bundle improves micro-F1 beyond an equal-volume control (+0.024 [0.013, 0.035]), also under campaign-grouped and publisher-held-out folds and in document-disjoint transfer to TRAM2; ATT&CK definition text recovers about two thirds of the gain over label-only training. Evidence-supervised span ranking improves overlap with expert evidence, which label metrics do not capture. With fully delivered prompts, retrieved demonstrations raise a smaller language model’s micro-F1 from 0.367 to 0.500 (labels) and 0.475 (full annotations; no reliable added benefit). Unlabeled chunks remain unverified candidate negatives.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.