Data Leakage in Speaker Credibility Features: A Systematic Analysis and Correction for the LIAR2 Fake News Benchmark

Abstract

The credibility history of the PolitiFact-based LIAR/LIAR2 fake news data sets causes label leakage and inflates reported performance. This work investigates two count-construction components: 1) the aggregation scope, which involves global aggregation over the whole data set before splitting, as done by LIAR/LIAR2 (Pre-Split Aggregation, PSA), and 2) self-inclusion, which uses the current row’s label in its history vector. We also define Within-Split Aggregation (WSA), per-split aggregation with self-exclusion, and Leak-Free Aggregation (LFA), the train-only counts without self-inclusion and zero vectors for speakers missing from the training data. The analysis is conducted via ablation and segmentation using the Fuzzy Deep Hybrid Network (FDHN) architecture. With self-exclusion applied, WSA is not significantly different from LFA in the full model (<inline-formula> <tex-math notation="LaTeX">$\Delta = -0.0014 \pm 0.0016$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$p = 0.12$ </tex-math></inline-formula>). Decomposition of PSA into two components (aggregation scope <inline-formula> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> self-exclusion) shows that leakage is caused mainly by self-inclusion, leading to a +0.1995 increase in the credibility-only Mean, while global aggregation scope contributes only +0.0028. In the complete model, PSA inflates the Mean by <inline-formula> <tex-math notation="LaTeX">$+0.0846 \pm 0.0038$ </tex-math></inline-formula>, while in all nine segmentation configurations PSA shows an increase between 4.0 and 18.2 percentage points compared to LFA. A second, smaller leak comes from the justification feature, whose text alone achieves a Mean of 0.5974. Any model trained on the released count columns is affected by the PSA construction and needs to be re-evaluated under LFA. Code and corrected data sets are publicly available.