Auditing unsupported claims in LLM-written intrusion alert explanations

Abstract

Large language models (LLMs) are now used to explain intrusion-detection alerts, yet an explanation can sound confident while stating things the evidence does not support. We divide such statements into two tiers. Tier 1 covers unsupported specifics and Tier 2 covers background knowledge that is absent from the evidence, so a single annotation yields both a strict and a lenient policy. We apply a frozen, versioned rule engine to 398 explanations that a Gemini model wrote from six flow features of CIC-IDS 2017 and nothing else; nine features were intended, but three were missing from the alert records of the earlier run. Under the strict policy, 92.5% of explanations (95% CI 89.7–95.0) contain at least one flagged sentence, and under the lenient policy 53.5% (48.5–58.3) do. Of 177 stated flow durations, 176 contradict the documented microsecond unit. At the sentence level, explanations of attack flows are flagged more often than those of normal flows (strict 24.3% vs 17.6%). In an audit of 25 explanations labelled by two AI annotators and adjudicated by the author (inter-annotator kappa 0.83), the rules never flag a sentence that the labels call grounded. They find, however, only 35% of the sentences the labels call unsupported (recall 0.35, 95% CI 0.28–0.44). We also report how an earlier version of this pipeline failed a reproduction, as several of its reported figures could not be recomputed from the saved files. The corpus comes from an earlier, uncontrolled run, the audit is small, and the labels are not an independent human gold standard. A controlled run has been specified but not executed.