Picking is Not Ranking, and Explanation Quality Has Many Dimensions: Lessons for Conversational Music Recommendation
Abstract
Conversational music recommendation requires predicting the next track a user wants and explaining the choice in natural language, scored by a composite of ranking accuracy, response diversity, and explanation quality. We present team FPMs_UMONS’s system for RecSys Challenge 2026 — hybrid four-channel retrieval, LLM reranking, and conversation-grounded response generation — and the lessons, negative results included, learned building it. Our central finding: a reranker’s two roles — choosing the track to recommend (picking) and ordering the remaining candidates (ranking) — are trained by different objectives. A listwise scoring head picks no better than single-pick generation but ranks the rest substantially better, the primary lever behind the leaderboard’s nDCG@20: the turns where the pick is wrong carry the entire gap. Further lessons: the provided multimodal embeddings help only once distilled into text descriptors the reranker can read; LLM judges select well but are unreliable for exact verification; part of the remaining error traces to the single-ground-truth task itself; and on established explanation-quality metrics and a user-study questionnaire administered by open-weight LLM judges, our explanations compare favorably with the dataset’s own reference replies. Built on small open-weight models (Qwen3-8B reranker), the system scored 0.4510 on the hidden test set, 9th of 18 academic teams.
