Tweet-Image-Caption conditioned diffusion model for multimodal Aspect-Based sentiment analysis

Abstract

Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to simultaneously extract aspects and predict their sentiment polarities from paired textual and visual content. Existing approaches typically formulate MABSA as either a classification or index generation task, which may limit the accuracy of aspect boundary localization, an essential component of end-to-end aspect-sentiment prediction. To address this issue, we propose TicCondDiffusion, a Tweet-Image-Caption conditioned diffusion model that reformulates MABSA as a denoising process over aspect boundary coordinates for aspect boundary localization and sentiment prediction, where the caption is an image-derived textual description used as auxiliary visual-semantic guidance. TicCondDiffusion consists of two main processes: a noising process and a conditioned denoising process. In the noising process, Gaussian noise is gradually added to ground-truth boundary coordinates over multiple timesteps to generate noisy boundary coordinates. In the denoising process, the denoising module, conditioned on Tweet-Image-Caption representations and timestep embeddings, recovers noisy boundary coordinates for boundary localization while simultaneously predicting the sentiment polarities of the corresponding aspects, with subsequent denoising steps providing further refinement. To effectively exploit multimodal information, TicCondDiffusion incorporates three specialized fusion modules: a Tweet-Image Fusion Module, a Tweet-Caption Fusion Module, and a Tweet Feature Fusion Module. Experimental results on Twitter-2015 and Twitter-2017 demonstrate that TicCondDiffusion achieves competitive performance compared with recent baseline approaches under the end-to-end setting. Experiments on the Political-Twitter dataset further show its adaptability to domains with different topical distributions. Additional analyses further validate the effectiveness and flexibility of the proposed framework. To the best of our knowledge, TicCondDiffusion is the first diffusion-based framework specifically designed for MABSA, offering a new direction for this task (The code is available on https://github.com/MEIMEIMEIMEIMEMEDA/TicCondDiffusion.).