Comparative Study of Traditional Deep Learning and Transformer-Based Architectures for Image-Text Multimodal Sentiment Analysis
Abstract
Social networks have become a powerful platform capable of shaping public opinion, which can subsequently influence and even shift policy decisions. The rapid dissemination of user-generated content, which often combines textual and visual information, makes understanding public sentiment increasingly complex. To address this challenge, sentiment analysis can be used to identify public attitudes. However, unimodal approaches are often insufficient to capture the full context of social media content. Therefore, this study adopts a multimodal sentiment analysis approach that integrates textual and visual modalities to better reflect real-world social media interactions. This research conducts a comparative analysis of deep learning models based on sequential and Transformer architectures to identify the most effective approach. The sequential model utilizes BiLSTM, while the Transformer-based approach employs ViLBERT, as well as a Cross-Encoder configuration that combines a Transformer for one modality and a Sequential for the other modality. The experimental results demonstrate that the full Transformer model, ViLBERT, achieves the highest overall robustness with a peak F1-Score of 0.6312, effectively outperforming the BiLSTM-ResNet baseline of 0.6014 and the Cross-Encoder BERT + BiLSTM-ResNet model of 0.6272. Furthermore, it is noteworthy that the hybrid Cross-Encoder configuration achieved the maximum performance in terms of accuracy, reaching a superior score of 0.7323. These results indicate that Transformer-based multimodal architectures are more effective in capturing complex sentiment patterns in social media data.