This paper presents the first user study to empirically assess the reliability of LLM-as-a-judge for evaluating CRS responses, and finds that LLM-based judges exhibit moderate positive alignment with human assessments and outperform all reference-based baselines.
Abstract
Conversational Recommendation Systems (CRS) aim to achieve two primary objectives: recommending relevant items and generating natural language responses. While recommendation accuracy is effectively measured by established ranking metrics, the evaluation of response generation poses a more fundamental challenge. Although human evaluation remains the gold standard, its cost and scalability constraints have motivated the adoption of LLM-as-a-judge as a promising proxy, whose alignment with human judgment in the context of CRS remains an open question. In this paper, we present the first user study to empirically assess the reliability of LLM-as-a-judge for evaluating CRS responses. We sample 20 multi-turn music recommendation sessions and generate candidate system responses using four instruction-tuned LLMs, inducing variance in response quality across model scales. We collect $n{=}400$ ratings from 20 domain-expert annotators, who evaluate each response across two dimensions: Personalization Quality and Explanation Quality. Through bootstrapped correlation analysis, we find that LLM-based judges exhibit moderate positive alignment with human assessments and outperform all reference-based baselines. Furthermore, we analyze how judge performance varies according to model scale and conditioning information, providing practical guidance for deploying LLM-as-a-judge.
The RecSys Challenge 2026 studies conversational music recommendation as a joint item recommendation and response generation problem: given a multi-turn dialogue, systems must retrieve relevant tracks from a large catalog and produce a grounded natural-language response. This paper presents the challenge task, dataset, evaluation protocol, and official results. Beyond the leaderboard, we analyze the 16 accepted systems through a common retrieve–rerank–generate framework and examine how recommendation performance varies across users, requests, and dialogue contexts. Strong systems commonly combine heterogeneous candidate sources and preserve source-specific evidence for learned reranking. Across the system papers and our organizer-side analysis, robust design also means 1) grounding cold-start retrieval in multi-turn conversation and item signals, 2) using intent detectors, and 3) modeling the full multi-turn context rather than the current query alone. We further identify limitations of the benchmark and evaluation protocol, including single-ground-truth relevance and teacher-forced evaluation of synthetic dialogues. Together, these findings provide practical guidance for future conversational recommender systems and shared evaluation efforts.
Seungheon Doh, Sergio Oramas, B. Sguerra et al.· Proceedings of the Workshop...· 0 citations
The RecSys Challenge 2026 Music-CRS (TalkPlay) task formalizes this as two coupled sub-problems: given dialogue history and user context, retrieve a ranked list of the top-20 tracks from the full, unrestricted catalog, and generate a response that justifies the recommendation while sustaining conversational coherence.
Simran Sundrani, Mohan Bhambhani· Proceedings of the Workshop...· 1 citation
Conversational music recommendation requires predicting the next track a user wants and explaining the choice in natural language, scored by a composite of ranking accuracy, response diversity, and explanation quality. We present team FPMs_UMONS’s system for RecSys Challenge 2026 — hybrid four-channel retrieval, LLM reranking, and conversation-grounded response generation — and the lessons, negative results included, learned building it. Our central finding: a reranker’s two roles — choosing the track to recommend (picking) and ordering the remaining candidates (ranking) — are trained by different objectives. A listwise scoring head picks no better than single-pick generation but ranks the rest substantially better, the primary lever behind the leaderboard’s nDCG@20: the turns where the pick is wrong carry the entire gap. Further lessons: the provided multimodal embeddings help only once distilled into text descriptors the reranker can read; LLM judges select well but are unreliable for exact verification; part of the remaining error traces to the single-ground-truth task itself; and on established explanation-quality metrics and a user-study questionnaire administered by open-weight LLM judges, our explanations compare favorably with the dataset’s own reference replies. Built on small open-weight models (Qwen3-8B reranker), the system scored 0.4510 on the hidden test set, 9th of 18 academic teams.
Maxime Manderlier, Fabian Lecron· Proceedings of the Workshop...· 1 citation
We describe team npatta01’s submission to the RecSys Challenge 2026 conversational music recommendation task. The pipeline extracts a typed conversation state with an LLM, gathers candidates from eleven retrieval branches over a unified track index, re-ranks them with a LambdaMART model, and uses the state to generate a response. On the final Blind-B leaderboard it scored 0.3811 composite (nDCG@20 0.2537, LLM-judge 3.30), ranking 29th of 40 teams. We then examine why. The development estimates we selected on were computed in-sample and overstated performance. The training conversations are LLM-generated, and on many turns we were unsure that the single ground-truth track matched the request — often it repeats the just-played artist after an explicit request for someone else. We re-judged the turns with LLM judges and release that relabeling; a model trained on it scored lower against the original labels, so the submission kept them. Failure cases from the submitted run show extracted constraints the pipeline could not enforce. Code, models, and a full reproduction bundle are publicly released.
Nidhin Pattaniyil, Semih Yagli, Tanwir Zaman· Proceedings of the Workshop...· 1 citation
This work proposes a new approach that quantifies the effectiveness of each interaction by the reduction in the assistant's uncertainty, measured via entropy over recommendations, to fine-tune the LLM, enabling strategic interaction generation.
Cedar Site Bai, Zhen-Yu Liao, Duan Li et al.· 0 citations
Traditional conversational recommenders entangle retrieval and response generation within a single text interface, so exact entity cues fade as the dialogue’s intent evolves, which compromises explanation credibility. We address this within the ACM RecSys Challenge 2026, which mandates both top-20 ranking and evidence-grounded response generation. This paper presents the third-place solution by team “swyoo” for the Blind-B industry track. We decouple retrieval and response into separate pipelines connected strictly via ranked tracks and metadata. Retrieval combines a hybrid lexical-dense pool for exact matching with a task-adapted pool driven by fine-tuned Qwen 8B adapters. Candidates are calibrated via LightGBM, then routed to an evidence-grounded propose-assign-select (PAS) framework to structure responses. This system also ranked second on the explanation-quality leaderboard in the final blind evaluation. Our findings demonstrate that: (i) isolating retrieval and response preserves both catalog cues and fluid intent; (ii) structuring generation via explicit evidence assignment supports this near-best-in-class explanation quality under the official LLM judge.
Sungwook Yoo, Se-Wook Yoo· Proceedings of the Workshop...· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
