TurkPhish v2: A Template-Composed Generation Matrix for Turkish Phishing E-Mail Corpora, with a Leakage Audit and a Detection Benchmark
TL;DR
This work introduces TurkPhish v2, a class-balanced corpus of 2653 Turkish e-mails built from a generation matrix crossing phishing themes, persuasion tactics, length bands, and stylistic registers; the matrix is encoded as LLM-ready prompts, while the released corpus is instantiated by a deterministic, anti-leakage template composer.
Abstract
Phishing remains the dominant initial-access vector in modern cyberattacks, yet Turkish-language resources for benchmarking phishing e-mail detectors are scarce and methodologically fragile. We first show that a publicly released Turkish phishing dataset—our own earlier one (v1; 7504 messages)—is pathological: after entity masking it collapses to seven phishing and one legitimate content templates, and 99.95% of sender addresses share one artificial pattern, so any classifier memorizes artifacts rather than phishing semantics. We introduce TurkPhish v2, a class-balanced corpus of 2653 Turkish e-mails built from a generation matrix crossing phishing themes, persuasion tactics, length bands, and stylistic registers; the matrix is encoded as LLM-ready prompts, while the released corpus is instantiated by a deterministic, anti-leakage template composer. Against the v1 pathologies, the corpus is clean: unique masked templates, no near-duplicates (maximum pairwise cosine 0.808), opening and punctuation views at chance (49.8–52.8%), and matched lengths (p = 0.999). We also report the residual these gates miss: class signal lives in a closed pool of intent sentences, so a zero-learning lookup rule reaches 0.928 macro F1 in distribution and 0.894 out of topic, outscoring five of the ten benchmarked detectors. In-distribution performance is therefore a ceiling artifact, not evidence of learned phishing semantics. Benchmarking ten detectors across classic machine learning, Turkish/multilingual transformers, and an instruction-tuned large language model, eight exceed 0.99 macro F1 in distribution and are statistically indistinguishable (McNemar, p > 0.05), whereas out of topic TF-IDF linear models lead (logistic regression 0.947, 95% CI [0.926, 0.966]; 0.959 under the body-only protocol we recommend to users of the corpus) and mBERT collapses (0.570). Because persuasion phrasing is shared across themes, this protocol measures transfer to unseen theme vocabulary rather than robustness to novel phrasing. The transformer deficit out of topic is, for two of three encoders, a thresholding rather than a ranking failure: mBERT retains 0.960 out-of-topic AUC while its recall at the default 0.5 rule falls to 0.263. The corpus, splits, generation framework, and the full leakage audit, including its negative results, are available for research use.