Leakage-Aware Cross-Dataset Evaluation of Prompt Injection Detection Using Classical Machine Learning and Transformer Models

Abstract

The widespread adoption of systems based on Large Language Models has made the reliable detection of prompt injection attacks a critical requirement. However, high performance achieved on training and test splits generated from the same data source does not guarantee that models can generalize to prompts from different sources. In this study, a leak-aware cross-dataset evaluation framework is presented to examine the robustness of classical machine learning and Transformer-based prompt injection detection models in the face of data source changes. During the data preparation process, empty records, duplicate prompts, conflicting labels, and text overlaps between datasets were checked. In this context, 38,184 duplicate records and 14 instances with conflicting labels were removed, and the 198 common prompts identified between the training and external test sets were removed only from the training set. Using WordHash and CharHash representations, SGD Logistic and Linear SVM models, as well as DistilBERT and DeBERTa-v3-small, were evaluated on an internal dataset consisting of 426,073 cleaned requests; the models were also tested on an independent dataset of 5,000 examples. While the models achieved performance in the range of approximately 0.997–1.000 in the internal evaluation, significant performance losses were observed in the external evaluation. DeBERTa-v3-small delivered the most balanced results, with an accuracy of 0.7360, a balanced accuracy of 0.7320, a Macro-F1 of 0.7284, and an attack sensitivity of 0.7119. The domain classifier achieved an ROC-AUC of 0.9770 hence indicating a remarkable shift in the distribution across data sources. The results show that internal validation results are not enough for prompt injection detection. Independent external validation, data leakage verification and domain shift analysis should be fundamental parts of reliable model evaluation.