Attention and Transformer based Multimodal Framework for Fine-Grained Emotion Analysis

Abstract

The increasing availability of social networking sites has ensured that memes have become a part of our daily lives to express our opinion through communication. Images or plain text are in contrast to multimodal content consisting of textual captions on images to explain emotions, opinions, and sarcasm in implicit and in some cases ambiguous ways. Identification and classification of these emotions are not a straightforward task but demand robust multimodal learning models that can glue visual as well as textual information together. This paper presents an end-to-end multimodal emotion analysis model based on the MEMotion dataset, integrating state-of-the-art deep learning structures, strict preprocessing, and extensive evaluation approaches to confront the difficulty of meme-based emotion detection. The proposed system is a formal multi-stage pipeline. Stage one delivers dataset refinement through cleaning, path correction, binarization of labels, and stratified splitting to balance classes in training and validation sets. Stage two is a hybrid approach in which ResNet-based convolutional networks learn discriminative visual features, while BERT encoders learn contextualized textual semantics. The features are then fused through dense projection layers and dropout-normalized to prevent overfitting. Stage three involves adaptive training mechanisms with the application of weighted cross-entropy, focal loss, and their hybrid forms for handling extreme class imbalance across multiple labels: humour, sarcasm, offensive, motivational, and general sentiment. Stage four enhances experimental robustness with gradient accumulation, learningrate warmup, early stopping, and misclassification grids. A broad set of per-task confusion matrix, aggregated confusion grid, ROC and precision-recall curve, loss–accuracy plot, and per-class accuracy plot outputs were generated to enable more in-depth model performance interpretability.