MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
Abstract
Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.
A Modality Dropout strategy is first introduced at the input stage to alleviate over-reliance on a sin-gle modality and improve robustness and the proposed Hierarchical Global-Local Interaction and Refinement framework for Multimodal Sentiment Analysis (HGLIR) is proposed.
yuanyuan zhou· Poster Volume 0007 The 2026...· 0 citations
This work proposes a novel framework that employs the pre-trained vision-language model BLIP (Bootstrapping Language-Image Pre-training) to generate descriptive image captions and introduces an aspect-guided soft prompt mechanism that enables dynamic interaction between aspect terms and multimodal features, thereby mitigating the effects of structural irregularities.
Multimodal Sentiment Analysis (MSA) aims to predict human sentiment by jointly modeling complementary information from textual, acoustic, and visual modalities. However, effectively exploiting heterogeneous multimodal features remains challenging due to semantic inconsistency, temporal misalignment, and noisy modality-specific representations. To address these issues, this paper proposes a Cross-Attention-based multimodal sentiment analysis framework that explicitly models inter-modal interactions through multi-directional cross-modal attention. Specifically, modality-specific features are first projected into a unified latent space via lightweight modality encoders, after which bidirectional cross-attention is employed to capture complementary dependencies among visual, acoustic, and textual modalities. To further enhance representation learning, multi-head attention and positional encoding mechanisms are incorporated to improve cross-modal interaction modeling and temporal structure awareness. Extensive experiments on the CMU-MOSI benchmark demonstrate that the proposed framework consistently outperforms conventional fusion baselines, including concatenation-based, additive, and self-attention-based fusion strategies. Comprehensive ablation studies further verify the effectiveness of each architectural component, while qualitative attention visualization confirms that the model learns interpretable attention patterns and focuses on semantically informative regions during multimodal fusion. These results indicate that the proposed method provides an effective and interpretable solution for multimodal sentiment analysis and offers useful insights for future attention-based multimodal fusion research.
DualScope is proposed, a novel model that combines a global-local fusion strategy with bidirectional image-text generation for semantically consistent data augmentation and introduces both label contrastive learning and data contrastive learning to align heterogeneous modalities and enhance model robustness.
Bing Zhang, Junteng Wang, Bin Sun et al.· Memetic Computing· 0 citations
A prototype-as-prompt framework that maps audio–visual representations into a fixed set of multimodal sentiment prototypes that are used as soft prompts to guide the LLM in performing MSA and introduces a sentiment-aware prototype learning that explicitly binds multimodal prototypes with sentiment semantics.
Xianbing Zhao, Lan Luo, Hengyang Lu et al.· 0 citations
MagXCL, a unified framework designed to improve multimodal integration through more effective interaction between verbal and non-verbal modalities, is proposed, demonstrating the effectiveness of combining AMag with CrossCL to produce more accurate and robust multimodal sentiment predictions.
What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduAug 31, 2026
With millions of users across the world, Julia has been used to conduct cutting-edge research and to design new drugs, jet engines, heat pumps, and more.