Tokenizing Numerical and Embedding Features for LLM RecSys

Zhe Xu Ankit Peshin Chiyu Zhang Fengtian Qi John C. S. Lui Anil Ramakrishna Justin Johnson Carl Hu Kaushik Rangadurai Luke Simon

Jul 2026 · arXiv.org · Vol abs/2607.10016 · 0 citations · 40 references

Computer Science

TL;DR

Experiments on three Amazon recommendation benchmarks show that soft-token fusion improves retrieval performance over LLM-based baselines, and that interaction-based fusion is more effective than direct concatenation of heterogeneous soft tokens.

Abstract

Large language models (LLMs) are increasingly used as backbone architectures for recommender systems because of their strong sequence modeling and representation learning capabilities. However, most LLM-based recommenders operate primarily on discrete textual tokens, whereas practical recommendation pipelines also rely on continuous numerical features and dense embedding features produced by upstream feature engineering or pretrained encoders. This mismatch limits the ability of LLM-based models to exploit fine-grained non-textual signals. We propose a soft-token fusion framework that maps numerical and embedding features into the LLM embedding space, allowing heterogeneous recommendation signals to be consumed through the standard token interface. We instantiate the framework in a shared-parameter LLM-based two-tower retrieval model and introduce an interaction-based fusion module that refines embedding and numerical soft tokens before they are inserted into the final LLM input. Experiments on three Amazon recommendation benchmarks show that soft-token fusion improves retrieval performance over LLM-based baselines, and that interaction-based fusion is more effective than direct concatenation of heterogeneous soft tokens.

DOI arXiv Semantic Scholar

View source

Similar papers

Jul 2026

LLM-Based Generative Retrieval for Snapchat Content Recommendation

The design and launch of SnapLGR, an LLM-based generative retrieval system for short-video recommendation at Snapchat shows that successful production SnapLGR requires joint design across representation learning, vocabulary grounding, and efficient training and serving.

Liam Collins, Jiwen Ren, Donald Loveland et al. · arXiv.org · 0 citations

Preprint Aug 2026

Empowering Compact LLMs with Fusion of Layer-wise Exits for Recommendation

The Fusion of Layer-wise Exits for Sequential Recommendation (FLEXRec), a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking and achieves state-of-the-art accuracy among competing methods while remaining highly efficient.

Xurong Liang, Tong Chen, Q. Nguyen et al. · 0 citations

#large language models Review Open access Sep 2026

PALRec: Large Language Model-Based Sequential Recommendation with Parameter-Preserving Augmentation

PALRec is proposed, a parameter-preserving augmentation framework that equips an LLM with recommendation capabilities while keeping its original parameters fixed and consistently outperforms fully fine-tuned counterparts in recommendation accuracy while preserving the LLM’s pre-trained knowledge.

Hyunsoo Na, Minseok Gang, Sang-goo Lee et al. · ACM Transactions on Informat... · 0 citations

Preprint Aug 2026

Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation

This work proposes a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space, and shows that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.

Donald Loveland, Liam Collins, B. Kumar et al. · 0 citations

Jul 2026

REPREC: Representation Driven Parameter-Efficient Recommendation System

REPREC is a lightweight framework that conditions a frozen LLM using compact user-level representations and consistently improves recommendation performance across different sequential encoders, LLM backbones, and user activity levels.

Harshini Kavuru, Dwipam Katariya, Giri Iyengar et al. · arXiv.org · 0 citations

Jul 2026

The Case Against Generation for Retrieval: Discriminative Language Models as Effective Retrievers

Large Language Models (LLMs) have emerged as powerful assets for recommender systems. However, deploying them as generative recommenders or zero-shot rankers at web-scale remains bottlenecked by prohibitive computational overhead and grounding challenges. In this paper, we revitalize the classic, highly efficient two-t...

Zhe Xu, Prachi Agrawal, Kavosh Asadi et al. · arXiv.org · 0 citations