Beyond Random-Split Accuracy: Duplicate-Safe and Crypto-Agile Evaluation of Anomaly Detection for Post-Quantum TLS

Abstract

Post-quantum cryptography changes the size, timing, and algorithmic context of Transport Layer Security (TLS) handshakes, creating a dynamic normal class for anomaly detectors. This study evaluates an assurance framework, rather than proposing a new classifier, on CIC-PQC_OAV v1: 40,010 sessions represented by 32 encrypted-metadata features. An audit finds 185 exact fingerprints shared across the supplied partitions, affecting 1594 sessions, plus one exact conflicting-label group. We compare the fixed split with five-seed, size-matched sample-stratified, exact-disjoint, raw round-3-disjoint, and IQR-normalized round-3-disjoint protocols, each separating fitting, probability calibration, threshold selection, conformal calibration, and testing. The fixed-split LightGBM F1 is 0.9013; controlled five-seed means cluster at 0.8872–0.8898, showing that its gap is not attributable solely to duplicate control. Condition-disjoint tests reveal heterogeneous anomaly transfer and false-positive rates of 0.9998, 0.9897, and 0.6107 for three unseen valid families. Isotonic calibration yields a Brier score of 0.0221±0.0013, while five-seed perturbations confirm sensitivity to timing masking and byte scaling. The results support layered, dataset-bounded evaluation combining fingerprint independence, condition holdouts, calibration, selective review, robustness, and explanation.