Toward Efficient Multimodal Recommendation: From Content Understanding to Personalized Generation

Abstract

Multimodal foundation models and generative AI are transforming recommender systems from ranking existing items into systems that can understand, combine, and generate personalized multimedia content. This keynote presents our recent exploration of this evolution across three important application domains: fashion, music, and micro-video. Technically, the talk follows three stages. First, for multimodal content understanding, we investigate how to learn effective representations from visual, textual, audio, symbolic, relational, and behavioral signals while avoiding costly end-to-end fine-tuning of large foundation models [2, 4, 6, 7, 9]. Second, for combinatorial modeling, we study how to capture compatibility and user preference within an enormous space of possible item combinations, using relational learning, compact discrete representations, and non-autoregressive generation to make the problem more tractable [3, 5, 8]. Third, for personalized content generation, we extend recommendation beyond selecting existing items toward creating fashion images, lyrics, music descriptions, and other user-tailored content, while efficiently reusing pretrained generative models [1, 10, 11, 12]. Together, these studies establish a technical progression from multimodal understanding, through combinatorial modeling, to personalized generation, and demonstrate its practical value across fashion, music, and micro-video. The talk concludes by discussing opportunities and open challenges in building multimodal recommender systems that jointly achieve effectiveness and efficiency.