Skip to content
Book Open access

Auditing Multimodal Claims: A Controlled Reproducibility Study of Graph-based Conversational Recommendation

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 27 references

Abstract

Multimodal conversational recommender systems (CRS) integrate dialogue, knowledge graphs, and visual signals. Recent work reports substantial gains over the knowledge-graph baseline KGSF on the INSPIRED dataset (R@10: 0.132→0.257). However, this pipeline couples ViT-based image embeddings with zero-fill handling for 23.7% of items missing poster data and augments training with LLM-generated synthetic dialogues, failing to isolate the visual modality. We present a controlled reproducibility study, maintaining the MSCRS backbone and training fixed while intervening on data and image-embedding inputs. To isolate item-specific visual correspondence from representation-level artifacts, we replace ViT embeddings with three distractor conditions: random Gaussian vectors, zero vectors, and shuffled real ViT embeddings. We also apply an artificial 23.7% missingness mask to ReDial and probe synthetic-dialogue training using a valid-best checkpoint protocol. Across INSPIRED and ReDial, the distractor conditions do not consistently separate from the original ViT condition; the multi-seed mean R@10 spread is only 0.007. Conversely, synthetic dialogue shows a directional paired difference (Δ R@10 = +0.031 \pm 0.020, Cohen's d_z=1.59). Furthermore, selection bias introduces a max-test versus valid-best gap of up to +0.035 R@10. These results do not imply that images are unhelpful, but rather that modality-specific visual effects are currently confounded by missing-node graph effects and augmentation artifacts. We propose a reproducibility-oriented audit procedure for multimodal CRS evaluation, covering distractor controls, missingness validation, and strict checkpoint selection.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.