Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 13272-13272· 0 citations
Abstract
For decades, search and recommendation systems have been optimized as distinct components within large-scale discovery platforms. The rise of generative AI is beginning to blur this boundary. At Spotify, we are exploring how large language models can evolve from tools that retrieve content into systems that reason over users, catalogs, and intent, while remaining steerable through natural language and user interaction. This talk presents lessons from deploying and studying generative retrieval and recommendation systems across Spotify's content ecosystem. I will describe how semantic identifiers enable language models to operate directly over large, heterogeneous catalogs, allowing search, recommendation, retrieval, explanation, and user understanding to be expressed within a common generative framework. I will discuss recent work on production-scale podcast discovery, language-steerable recommendation, and the NEO framework for unifying search, recommendation, and reasoning across multiple content types. These systems demonstrate how grounding language models in catalog entities and user behavior can improve discovery while preserving the flexibility of natural-language interaction. More broadly, they suggest a path toward discovery systems in which retrieval, recommendation, and reasoning are no longer separate stages, but capabilities of a shared generative model. Beyond model frameworks, I will discuss the emerging challenges of alignment and evaluation in discovery systems. Unlike traditional retrieval problems, generative recommendation often has many valid answers. I will present approaches for learning from large-scale behavioral signals, preference-aware optimization, and profile-aware LLM-as-a-judge evaluation, along with lessons from online experimentation at Spotify. These experiences suggest that future discovery systems will require new forms of personalization, controllability, and evaluation that extend beyond conventional ranking metrics. I will conclude with a research agenda for generative discovery systems, including language-steerable interfaces, unified retrieval-and-reasoning models, preference-aligned generation, and evaluation frameworks designed to measure user-specific relevance at scale. As search, recommendation, and conversational AI continue to converge, these directions point toward a new generation of discovery systems that can understand intent, reason over large catalogs, and help users navigate increasingly complex information spaces.
This work proposes GUIDER (Generative User Interest Discovery & Explicit Reasoning), a framework that fundamentally decouples intent planning from item matching by reformulating sequential modeling within a rigorous closed-set semantic interest space.
Jin-Ke Wu, Ying-Hao Wu, Shuchang Liu et al.· Proceedings of the 32nd ACM...· 0 citations
A literature-based architectural framework for reliable knowledge retrieval systems that separates external knowledge management from LLM-based reasoning and generation is developed and indicates that reliable LLM deployment should be treated as an end-to-end architectural problem rather than solely a model-performance problem.
Bharat Kumar Reddy Karumuri· International Journal of Eng...· 0 citations
Large language models (LLMs) have transformed AI, yet they remain fundamentally limited by hallucination, unverifiable reasoning, and shallow evidence grounding. We argue that structure mining-rooted in decades of KDD research on taxonomy induction, ontology design, entity typing, and knowledge graph construction-is the key to overcoming these limitations. This tutorial presents a unified vision in which structuring serves as the enabling foundation for three pillars of next-generation LLM systems: (1) Structured Retrieval, where organizing corpora into ontology-guided multidimensional representations enables SQL-like queries that achieve substantially more precise and complete retrieval than similarity-based approaches; (2) Structured Reasoning, where grounding each inference step in typed, graph-structured evidence transforms opaque generation into auditable, verifiable reasoning chains; and (3) Structured Agent Memory, where multi-dimensional memory architectures bridge external corpus knowledge and experiential agent knowledge through a mutually enriching dual-memory design. Across all three pillars, we highlight how the cooperative interplay between classical KDD techniques and modern LLMs-where KDD defines structural schemas and quality constraints while LLMs execute flexible extraction and reasoning-creates systems that are more reliable, interpretable, and faithful. The tutorial covers both foundational methods and the latest advances (2024--2026), and concludes with open problems and future research directions at the intersection of data mining and LLMs.
Pengcheng Jiang, Jiashuo Sun, Wonbin Kweon et al.· Proceedings of the 32nd ACM...· 0 citations
GEM is presented, a generative embedding model that augments retrieval through its own knowledge by explicitly reasoning about user intent and relevance criteria, and its generative nature allows test-time compute scaling via prompting to further enhance retrieval performance.
Large language models can improve recommendation quality by reasoning explicitly over user history and candidate items - for example, extracting a user's preferences or explaining why one item fits better than another - rather than mapping history directly to a ranked list. This reasoning, however, is expensive to repeat on every ranking request and, once produced, is typically consumed once and discarded, leaving it neither reusable across future requests nor easy to inspect or correct as user tastes drift. Our insight is that reasoning does not need to be regenerated at every call if it can instead be compressed once into a compact, structured memory that a lightweight model retrieves from. We propose rEDMRec, which distills a teacher LLM's reasoning into four typed, editable experience channels - long-term preference, short-term context, item-perception, and counterfactual hard-negative comparisons - maintained by an LLM memory controller that performs Add/Delete/Modify/Keep operations and refines entries via K-agent debate. A lightweight student LLM then ranks candidates purely by retrieving from this memory, without invoking the teacher again, decoupling online inference cost from reasoning depth. Across ML-1M, Amazon Beauty, and Steam and ten student backbones, rEDMRec improves HR@1 over zero-shot, few-shot, and RAG on every backbone, and over GraphRAG on most backbones, with Impv up to 13.3% vs. the second-best baseline on ML-1M. Channel ablations show that short-term context is the only channel that helps consistently across capacity tiers, whereas long-term, item-perception, and counterfactual contributions are capacity-dependent (and can reverse on the strongest students); debate-based memory optimization lowers bank duplication by 7.4 percentage points while raising downstream HR@1 by up to +0.029 over six optimization epochs.
M. H. Nguyen, Tung Le, Huy-Tien Nguyen· 0 citations
Traditional search engine returns ranked lists for humans to interpret. Retrieval Augmented Generation pipelines go further, feeding retrieved context directly into large language models to synthesize knowledge rather than simply surface it. This study addresses a focused question: When the generative layer is held constant, how do distinct retrieval architectures perform across diverse query types? We fix the LLM and chunking strategy while independently varying the retrieval backend and embedding model across three configurations: Memory Resident Specialist (MRS), Industrial Semantic Index (ISI), and Agile Hybrid Scaler (AHS) evaluated over a six-tier query taxonomy. Our results demonstrate that retrieval quality acts as a rigid performance ceiling for generative correctness. Notably, the "Efficiency Paradox" emerged: the AHS configuration, leveraging lightweight MiniLM embeddings with Pinecone cloud indexing, achieved parity with proprietary high-dimensional configurations on complex logical reasoning tasks. However, all three architectures share a critical weakness for conversational queries. Multi-turn contexts revealed a systemic failure stemming from the absence of anaphora resolution in naive RAG pipelines. These findings suggest that for RAG applications, strategic database indexing is a more critical performance lever than embedding parameter scale: architectural retrieval decisions matter more than model size.
Assmaa Moutaoukkil, Ali El Mezouary, A. Idarrou et al.· EPJ Web of Conferences· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
