Instruction-Tuned LLMs via Bayesian Mixture Model Preference Priors

Abstract

Large language models (LLMs) deployed as interactive recommendation agents must adapt to user preferences across dialogue rounds without access to ground-truth reward functions. Existing Bayesian-prompting approaches assume a uniform prior over user types, discarding population structure and weakening cold-start performance. We propose an empirical Bayes framework that fits a Dirichlet Compound Negative Multinomial mixture with Feature Saliency (DCNM-FS) to interaction logs without supervision, recovering interpretable user-type archetypes with per-feature saliency scores. At inference time, a closed-form posterior over user types is updated after each choice and injected into the LLM context, while a session-local exponential moving average further personalizes the profile. On a synthetic flight-search dataset and the WebShop e-commerce benchmark, DCNM-FS context improves recommendation accuracy by 30%, with the largest gains in later rounds, where the structured posterior adapts to individual observations.