Adaptive Fusion-Driven Multimodal Sentiment Analysis Using U-Net++, Capsule Network, and Transformer-Enhanced Extreme Learning

Abstract

Multimodal social media data contain textual, visual, and emoji information that can provide complementary cues for sentiment analysis. However, combining these modalities remains challenging because they differ in representation and may contain redundant or noisy information. This study presents an Adaptive Fusion-Driven Multimodal Sentiment Analysis Framework that combines U-Net++ with self-attention, a Capsule Network encoder-decoder, a Cross-Modal Transformer, and a Transformer-Enhanced Extreme Learning Machine (T-ELM). U-Net++ with self-attention extracts contextual features, while the Capsule Network captures hierarchical relationships among the extracted representations. The Cross-Modal Transformer then models interactions between the text, image, and emoji modalities before classification by T-ELM. The trimodal experiments use 9,200 samples selected from the Social Media MultiSent Dataset (SM-MSD), with positive, neutral, and negative sentiment polarity as the target classes. Using a 70% training, 15% validation, and 15% test split, the proposed framework achieves competitive sentiment classification performance across the evaluated datasets. The results show that the proposed fusion framework effectively combines complementary multimodal information while keeping the final classification stage lightweight.