Auditing Multimodal Claims: A Controlled Reproducibility Study of Graph-based Conversational Recommendation
Abstract
Multimodal conversational recommender systems (CRS) integrate dialogue, knowledge graphs, and visual signals. Recent work reports substantial gains over the knowledge-graph baseline KGSF on the INSPIRED dataset (R@10: 0.132→0.257). However, this pipeline couples ViT-based image embeddings with zero-fill handling for 23.7% of items missing poster data and augments training with LLM-generated synthetic dialogues, failing to isolate the visual modality. We present a controlled reproducibility study, maintaining the MSCRS backbone and training fixed while intervening on data and image-embedding inputs. To isolate item-specific visual correspondence from representation-level artifacts, we replace ViT embeddings with three distractor conditions: random Gaussian vectors, zero vectors, and shuffled real ViT embeddings. We also apply an artificial 23.7% missingness mask to ReDial and probe synthetic-dialogue training using a valid-best checkpoint protocol. Across INSPIRED and ReDial, the distractor conditions do not consistently separate from the original ViT condition; the multi-seed mean R@10 spread is only 0.007. Conversely, synthetic dialogue shows a directional paired difference (Δ R@10 = +0.031 \pm 0.020, Cohen's d_z=1.59). Furthermore, selection bias introduces a max-test versus valid-best gap of up to +0.035 R@10. These results do not imply that images are unhelpful, but rather that modality-specific visual effects are currently confounded by missing-node graph effects and augmentation artifacts. We propose a reproducibility-oriented audit procedure for multimodal CRS evaluation, covering distractor controls, missingness validation, and strict checkpoint selection.