Does It Generalize? A Cross-Dataset Study of Graph-Based Insider Threat Detection Beyond CERT

Abstract

Graph-based insider threat detection methods increasingly report strong performance by modelling relationships among users, hosts, resources and security events. However, many studies still evaluate within a single benchmark family—usually CERT—or report separate within-dataset results, so it is unclear whether the learned representations capture transferable insider behaviour or dataset-specific artefacts. This study presents, to the best of our knowledge, the first systematic zero-shot cross-dataset benchmark for insider threat detection across CERT, SPEDIA and LANL. In this setting, no labelled or unlabelled target observations are used during training. Events are harmonised to a common user-day prediction unit with label-independent event sampling, and in-distribution performance is controlled with both chronological and user-disjoint splits. Within-domain performance of a Random Forest reference model was heterogeneous (user-disjoint PR-AUC from 0.004 on LANL to 0.99 on SPEDIA). Across evaluable targets, its mean in-distribution PR-AUC was 0.542, compared with a mean off-diagonal zero-shot PR-AUC of 0.133, yielding a generalization gap of 0.409. A memory-based temporal graph neural network without explicit time encoding was compared with this reference on two transfers into SPEDIA using paired seed-level bootstrap intervals. Representation effects were source-dependent: the temporal graph model outperformed the Random Forest for CERT r4.2→SPEDIA (∆PR = +0.123, 95% CI [+0.075, +0.169]), whereas the Random Forest out-performed the graph model for CERT r5.2→SPEDIA (∆PR = −0.031, 95% CI [−0.061, −0.012]). The results therefore do not support a general advantage of graph representations. They show that strong single-dataset performance is insufficient evidence of cross-dataset generalization.