Personalized Context-Conditioned Tokenization for Generative Recommendation

Abstract

Generative recommendation (GR) represents user actions as sequences of discrete tokens (often referred to as semantic IDs) and predict future interactions through autoregressive generation. This paradigm offers advantages such as compact representations, scalability, and the potential to unify retrieval and ranking. However, existing tokenization schemes are typically static and agnostic to user-specific context. In most cases, semantic IDs are derived solely from item attributes, implicitly assuming a shared notion of item similarity across all users. Under autoregressive decoding, tokens with identical prefixes tend to receive similar probabilities, meaning that a fixed tokenization enforces a universal similarity structure that may not reflect individual user perspectives. In reality, users may interpret the same item in fundamentally different ways depending on their preferences and intents. To better capture such diversity, we introduce a context-conditioned tokenizer that integrates a user's historical interactions into the tokenization process. This approach allows a single item to be mapped to multiple semantic ID sequences under different user contexts, enabling the model to represent heterogeneous interpretations and produce more personalized recommendations. Empirical results on three public benchmarks show that our method achieves up to 11.44% improvement in NDCG@10 compared with non-personalized tokenization approaches. The implementation is available at https://github.com/YoungZ365/Pctx.