Multimodal Fake News Detection via Hybrid External Knowledge and Offline LLM Supervision

Abstract

Multimodal fake news detection requires both finegrained text-image alignment and external factual context. This paper presents a three-stage framework centered on modality-specific global-token cross-attention. The proposed attention mechanism independently parameterizes text-to-image and image-to-text projections to accommodate the asymmetric semantic structures of textual tokens and visual patches. An offline large language model is used only on the training split to produce veracity-related supervision and entity annotations. A local multi-task student reproduces these outputs during validation and testing, eliminating test-time cloud dependency. The predicted entities provide bridges to a language-adaptive hybrid knowledge backend consisting of Wikipedia, Wikidata, and Baidu Baike. Retrieved triples and evidence are encoded through graph attention and integrated with BERT and ViT representations. A stratified analysis of 900 GossipCop training samples evaluates annotation errors across classes, topics, and text lengths. On GossipCop, the complete model achieves 90.46% accuracy, 0.808 Macro-F1, and 0.664 Fake-F1. Compared with shared-projection GTA, modality-specific parameterization improves accuracy by 1.12 percentage points and Macro-F1 by 0.022. The framework further obtains 91.58% accuracy on Weibo and 85.06% accuracy under PolitiFact fine-tuning. Comparisons with knowledge-enhanced, CLIP-based, and multimodal-LLM baselines demonstrate the complementary value of cross-modal alignment, structured factual context, and offline LLM-derived supervision.