Mix-Attention for Multimodal Large Language Model-Enhanced Sequential Recommendation

Abstract

Sequential recommendation aims to predict the next item that may interest users by modeling their historical item sequence. Recently, multimodal sequential recommendation has achieved promising results in capturing user preferences by incorporating rich modality information. However, existing frameworks suffer from limitations in two key components: (1) the modality encoder used to obtain item representations is typically pre-trained on general-domain data and cannot capture recommendation-oriented knowledge. (2) The sequential recommender to model the multimodal representation sequence struggles to capture both modal-independent characteristics and cross-modal interactions simultaneously. In this paper, we propose Mix-Attention for Multimodal Large Language Model-enhanced SEquential Recommendation, i.e., MASER. Specifically, we incorporate essential item relation knowledge of recommendation scenarios into the Multimodal Large Language Model (MLLM) through instruction-tuning, enabling it to capture recommendation-relevant information during modality encoding. In addition, we propose a Mix-Attention-based multimodal user preference modeling approach to jointly capture modal-specific item transition patterns and item interactions across modalities. Experiments conducted on multiple public datasets show that our proposed method significantly outperforms existing multimodal sequential recommendation approaches. Further analysis confirms the effectiveness of incorporating recommendation knowledge into modality encoding and elaborate user preference modeling based on the Mix-Attention mechanism.