Explainable Fake News Detection Using Transformers: A Comparative Study of SHAP and Integrated Gradients with Cross-Dataset Generalization

Abstract

As fake news continues to proliferate at an accelerating pace via digital media, automated fake news detection has become an urgent research challenge. We share a complete pipeline for explainable fake news detection from the WELFake dataset, consisting of 72,134 articles from 4 sources, that is reproducible. Four models are compared on equal footing: Logistic Regression with TF-IDF features, Linear Support Vector Machine, Bidirectional LSTM, and DistilBERT transformer fine-tuned with a 3-seed protocol. DistilBERT finishes first on every metric, with a macro F1 of 0.9807 ± 0.0008, an accuracy of 98.08%, a ROC-AUC of 0.9979, and an MCC of 0.9616, and the margin over the classical baselines is statistically significant, with a McNemar χ² of 61.99 and a p-value of 3.45×10⁻¹⁵. Its mistakes, though, are one-sided. Out of 132 remaining errors, 95 are actually articles that were misclassified as fake and only 37 are fake articles that were misclassified as true; thus, the most frequent error of the algorithm is to block legitimate articles. Then, directly test the generalization by transferring the trained models to the GonzaloA benchmark. Here, DistilBERT suffers 3.86 pp macro F1, whereas the TF-IDF models maintain a similar score, which encapsulates the difference between in-distribution memorization and true domain-invariant generalization. When applied to token-level predictions of the transformer, IG is also much more faithful, with an AUCDel of 0.1667 compared to 0.0001 for SHAP, and runs substantially faster, taking 2.36 seconds per sample against 4.75 seconds per sample for SHAP, while their Jaccard agreement is very low at 0.0231 ± 0.0515, meaning that they read the model in complementary ways. The wire service dateline tokens "Washington" and "Reuters" are the model's strongest evidence for real news; tellingly, SHAP exposes outright source leakage. Lastly, using temperature scaling with an optimal temperature of 0.8321 reduces the Expected Calibration Error from 0.0376 to 0.0143, amounting to a 62.0% relative reduction, at no additional cost of accuracy.