Deep Learning Methods for Multimodal Fake News Classification Combining Textual and Visual Information
TL;DR
The authors suggest a computationally efficient multimodal deep learning framework using Bidirectional Encoder Representations of Transformers (BERT) to extract textual features and convolutional neural networks to learn visual representations that offers a computational scaling alternative to attention-based models, which is computationally expensive.
Abstract
The multimodal nature of fake news propagating in the social media has necessitated the need to have strong detection systems that can detect textual and visual discrepancies. In this work, the authors suggest a computationally efficient multimodal deep learning framework using Bidirectional Encoder Representations of Transformers (BERT) to extract textual features and convolutional neural networks (ResNet50, MobileNet and VGG16) to learn visual representations. It uses a feature-level fusion approach that involves the integration of contextual text embeddings and deep visual features, and a softmax-based classification layer. The Fakeddit dataset is experimented on a six-class classification configuration, with unequal data distribution. The suggested multimodal model (BERT + ResNet50) is more effective with an accuracy of 94.7% and a macro F1-score of 0.91, and recall, as compared to unimodal baselines. Image-only models are performing moderately (75-78% accuracy) and the text-only BERT model is at 88.3 percent accuracy which shows the significance of multimodal integration. The findings show that feature-level fusion is effective to capture cross-modal discrepancies, minimizing false positives and enhancing generalization. The presented framework offers a computational scaling alternative to attention-based models, which is computationally expensive, and forms a solid basis in future multimodal misinformation detection studies.