A PRISMA-Derived Deterministic Scoring Framework for Intrusion Detection Dataset Evaluation

Abstract

Dataset selection is one of the most consequential methodological decisions in cybersecurity intrusion detection research. However, it is routinely made based on citation frequency, historical convention, and availability rather than principled evaluation. This allows benchmarks with well-documented limitations to persist, obscuring whether reported performance gains reflect genuine advances or artefacts of data construction. This study addresses that gap by introducing a deterministic, property-based scoring framework. The framework is grounded in a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-guided systematic literature review of 112 peer-reviewed studies published between 1998 and 2024, identified through scholarly database searches conducted through June 2026. It synthesises recurring dataset characteristics into Essential and General property sets and operationalises them into six auditable evaluation categories. Applied to 27 representative datasets, the framework exposes systematic trade-offs that aggregate rankings can miss. It reveals high provenance fidelity and broad attack coverage are structurally compensatory. It also shows that legacy benchmarks occupy a different suitability tier than their citation dominance implies, and that evaluation readiness, not documentation, has the most room for improvement across the current corpus. The result is a reproducible, model-agnostic infrastructure that converts dataset selection from an implicit habit into an explicit, evidence-driven methodological decision. This establishes dataset evaluation as a step deserving the same rigor as algorithm design and experimental protocol specification.