Personalizing Large Language Models with User Profile Memory

Abstract

Large language models (LLMs) are increasingly used in personalized applications, where responses must align with individual user preferences, histories, and profiles. A common approach is to inject user information into the prompt at inference time. However, existing methods typically rely on flat profile representations and similarity-based Top-K retrieval, which often introduce noisy or marginally relevant context under limited prompt budgets. In this paper, we propose PersMem, a user profile memory framework for LLM personalization that addresses three key questions: what user information to store, how to organize it, and how to use it effectively during generation. First, we construct a structured user profile memory by extracting entities and concepts from user profiles. Second, we organize the memory into a hierarchical structure that separates short-term interactions from long-term knowledge, enabling scalable and coarse-to-fine retrieval. Third, we introduce a learning-based memory gating mechanism that adaptively selects long-term memory units under prompt budget constraints and directly optimizes answer quality through evaluated signals. Extensive experiments on multiple personalization tasks demonstrate that our approach consistently improves personalization effectiveness while reducing prompt length, compared with existing prompt-based and retrieval-based baselines.