A real-time multimodal digital human interaction system via cross-attention fusion of LLaMA and CLIP models

Abstract

To address the core limitations of existing digital human systems, including shallow multimodal semantic understanding, weak dynamic personalization, and poor end-to-end real-time performance, this paper proposes a real-time multimodal digital human interaction system via cross-attention fusion of LLaMA (Large Language Model Meta AI) and CLIP (Contrastive Language-Image Pre-training). The main contribution is a token-level cross-attention module that aligns CLIP visual features with LLaMA language states and an interest-decay LSTM that updates user profiles after each interaction; together, these mechanisms improve multimodal matching, intention recognition, and personalized recommendation while retaining real-time inference. Experiments are conducted on MSVD and MSR-VTT using their official train/validation/test partitions, while the 1,000 annotated interaction samples use an independent 8:1:1 split; AdamW is used with a learning rate of 2e−5 and a batch size of 32. The proposed model achieves 90.3% image-text matching accuracy, 87.6% intention recognition accuracy, and 82.5% personalized recommendation accuracy. In subjective evaluation, it obtains 4.6/5 user satisfaction and 4.5/5 dialogue naturalness, significantly outperforming text-driven systems. The batch-size−1 inference latency is 180 ms on an A100 GPU, meeting the real-time interaction requirement.