HAFT: Hierarchical Audio-Enhanced Fusion Transformer for Efficient Multimodal Sentiment Analysis

MULTI-MODAL TRANSFORMER ARCHITECTURE WITH CROSS-ATTENTION FUSION FOR ROBUST AUDIO-VISUAL SENTIMENT ANALYSIS

Multimodal sentiment analysis (MSA) has gained significant attention due to its ability to integrate heterogeneous information from audio, visual, and textual modalities. However, existing transformer-based fusion methods often suffer from reduced robustness when one or more modalities are corrupted or partially unavailable. This paper presents a Multi-Modal Transformer Architecture with Cross-Attention Fusion (MMT-CAF) for robust audio-visual sentiment analysis. The proposed framework combines modality-specific transformer encoders, bidirectional cross-attention, and a reliability-aware fusion mechanism that dynamically adjusts the contribution of each modality according to its estimated reliability. The framework was evaluated on the CMU-MOSI and CMU-MOSEI benchmark datasets and compared with representative transformer-based methods, including Adaptive Modality Weighting, RAFT, and CITN-DAF. Experimental results demonstrate that MMT-CAF achieves superior sentiment classification performance while maintaining higher robustness under noisy audio, visual occlusion, and missing-modality scenarios. Ablation studies further confirm the effectiveness of the proposed cross-attention and reliability-aware fusion modules in improving multimodal representation learning. The proposed architecture provides an effective and interpretable framework for robust multimodal sentiment analysis and offers a promising foundation for real-world affective computing applications.

Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis

MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.

GICA: gate-controlled information bottleneck cross-modal attention network

XSentiFusionNet: An Explainable Cross-Modal Attention Framework For Audio-Visual Sentiment Analysis Using Hybrid Deep Learning

The exponential growth of audio-visual content created through social media, communication platforms as well as human-computer interaction has led to a need for effective multimodal sentiment analysis. Most of the multimodal frameworks have limitations in cross-modal interaction modeling, fusion strategy adaptability, interpretability, and robustness to noise. This paper introduces XSentiFusionNet, an end-to-end explainable multimodal framework for audio-visual sentiment analysis that incorporates cross-modal transformer-attention, reliability-aware adaptive fusion, and XAI. Our approach uses CNNs, BiLSTMs, and ViTs to capture emotional features while adaptively combining acoustic and visual modalities based on their reliability levels. Model transparency is achieved by integrating SHAP, LIME, Grad-CAM, and attention mechanisms. Comprehensive experiments were performed on CMU-MOSEI, MELD, and RAVDESS benchmark datasets using comparative analysis, ablation studies, cross-dataset generalization experiments, confusion matrix analysis, explainability, and robustness to noise experiments. Our framework achieved an accuracy of 94.82%, F1-score of 94.11% and a ROC-AUC score of 96.04% on the benchmark datasets performing better than existing approaches such as CNN-LSTM, Transformer Fusion, and Multimodal BERT. Additionally, the framework showed higher robustness to noise and generalization ability on different multimodal datasets while the XAI module demonstrated interpretability by highlighting key speech and facial features used for predictions. These results show that XSentiFusionNet is a reliable and efficient framework for audio-visual sentiment analysis and can be used in real-world multimodal processing and affective computing scenarios.

MagXCL: enhanced multimodal adaptation gate and cross-modal contrastive learning for multimodal sentiment analysis

MagXCL, a unified framework designed to improve multimodal integration through more effective interaction between verbal and non-verbal modalities, is proposed, demonstrating the effectiveness of combining AMag with CrossCL to produce more accurate and robust multimodal sentiment predictions.

Learning Adaptive Cross-Modal Interactions for Multimodal Sentiment Analysis

Multimodal sentiment analysis aims to integrate heterogeneous textual, visual, and acoustic information for effective emotion understanding. However, existing methods often suffer from insufficient cross-modal interaction modeling, limited adaptability in multimodal fusion, and inadequate suppression of modality-specific noise under complex conversational scenarios. To address these challenges, this paper proposes a framework for learning adaptive cross-modal interactions for multimodal sentiment analysis. The proposed framework consists of three stages: modality-aware preprocessing, heterogeneous representation learning, and adaptive multimodal fusion. First, a unified preprocessing strategy is designed to improve cross-modal consistency through textual normalization, speaker-aware visual alignment, and utterance-level acoustic representation enhancement. Second, modality-specific encoders are constructed to capture complementary semantic, spatial, and utterance-level acoustic characteristics from textual, visual, and acoustic modalities, respectively. Third, an adaptive fusion framework is introduced to explicitly model cross-modal interactions, dynamically estimate the importance of different modality combinations, and further calibrate discriminative feature channels through channel attention. By jointly performing modality-level interaction learning and channel-wise feature refinement, the proposed framework effectively enhances multimodal representation capability for sentiment classification. Extensive experiments conducted on the CMU-MOSI and MELD benchmark datasets demonstrate that our framework consistently outperforms previous methods. In particular, the proposed model achieves 90.27% accuracy and 90.26% F1-score on CMU-MOSI, together with 66.57% accuracy and 66.21% F1-score on MELD. Additional ablation studies and qualitative analyses further validate the effectiveness of the proposed preprocessing strategy, modality-specific representation learning, and adaptive fusion mechanism.