Jun 2026· 2026 IEEE 2nd International Conference on Electronics, Energy Systems and Power Engineering (EESPE)· pp. 930-935· 0 citations· 14 references
Abstract
Multimodal Sentiment Analysis (MSA) aims to recognize affective information by jointly exploiting signals from multiple modalities. Among textual, acoustic, and visual inputs, the textual modality usually conveys the primary semantic information associated with sentiment, whereas the other two modalities provide complementary nonverbal evidence. Based on this observation, this paper presents the Text-Guided Joint Interaction Network (TJINet), which promotes sufficient interaction between acoustic and visual information before introducing textual guidance. First, the features of each modality are independently encoded and transformed into compact representations. Next, the Gated Cross-Attention Joint Audio-Visual Interaction (JAVI-GCA) module performs bidirectional interaction between the acoustic and visual modalities and combines their complementary information into a joint audio-visual representation. Subsequently, the Language-guided Fusion Layer employs textual features as queries to selectively retrieve sentiment-related information from the previously fused audio-visual representation. The resulting multimodal representation is finally used to generate sentiment predictions. Experiments conducted on the CMU-MOSI and CH-SIMS datasets demonstrate that TJINet delivers better overall performance than several existing advanced methods.
Multimodal sentiment analysis aims to integrate heterogeneous textual, visual, and acoustic information for effective emotion understanding. However, existing methods often suffer from insufficient cross-modal interaction modeling, limited adaptability in multimodal fusion, and inadequate suppression of modality-specific noise under complex conversational scenarios. To address these challenges, this paper proposes a framework for learning adaptive cross-modal interactions for multimodal sentiment analysis. The proposed framework consists of three stages: modality-aware preprocessing, heterogeneous representation learning, and adaptive multimodal fusion. First, a unified preprocessing strategy is designed to improve cross-modal consistency through textual normalization, speaker-aware visual alignment, and utterance-level acoustic representation enhancement. Second, modality-specific encoders are constructed to capture complementary semantic, spatial, and utterance-level acoustic characteristics from textual, visual, and acoustic modalities, respectively. Third, an adaptive fusion framework is introduced to explicitly model cross-modal interactions, dynamically estimate the importance of different modality combinations, and further calibrate discriminative feature channels through channel attention. By jointly performing modality-level interaction learning and channel-wise feature refinement, the proposed framework effectively enhances multimodal representation capability for sentiment classification. Extensive experiments conducted on the CMU-MOSI and MELD benchmark datasets demonstrate that our framework consistently outperforms previous methods. In particular, the proposed model achieves 90.27% accuracy and 90.26% F1-score on CMU-MOSI, together with 66.57% accuracy and 66.21% F1-score on MELD. Additional ablation studies and qualitative analyses further validate the effectiveness of the proposed preprocessing strategy, modality-specific representation learning, and adaptive fusion mechanism.
Chuhan Cheng, Hangcheng Wu, Junqiao Wang et al.· International Conference on...· 0 citations
The high rate of social media content development causes an increase in multimodal data, such that modeling relationships between visual and textual data is challenging. Nevertheless, most of the available methods cannot capture fine-grained text-to-visual or visual-to-text interaction, resulting in lower sentiment performance. A Contrastive Bidirectional Cross-Modal Attention (C-BCMA) model is presented to enhance the correspondence of textual and visual representations by acquiring a common latent space. An attention method inspired by CLIP is utilized to produce robust cross-modal latent features to enhance their joint representation. Textual features are derived using ALBERT, whereas EfficientNet-B2 is applied to obtain visual representations. Interactions between modalities are learned using a multi-head attention mechanism. Textual and visual information is handled jointly during learning. This helps reduce gaps between the two modalities. This enables the model to process various semantic cues at once. Contrastive learning is used in the model to align similar text-image pairs and to separate unrelated text-image pairs so that better multimodal representations are achieved. The model has a better performance than baseline approaches on both single and multiple annotation versions of MVSA datasets. It achieves better performance across various evaluation metrics. Less obvious expressions like sarcasm and implicit sentiment are handled more effectively in this work, improving interpretation in multimodal sentiment analysis of social media data.
Prashant Adakane, Amit Gaikwad· international journal of eng...· 0 citations
MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
Shanshan Lin, Yuesheng Wu, Chao Chen et al.· 0 citations
MagXCL, a unified framework designed to improve multimodal integration through more effective interaction between verbal and non-verbal modalities, is proposed, demonstrating the effectiveness of combining AMag with CrossCL to produce more accurate and robust multimodal sentiment predictions.