SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching

Abstract

Modern short-video recommenders must exploit ultra-long user histories—which can reach hundreds of thousands or even millions of interactions per user—but are constrained by strict latency and training-throughput budgets. As a result, production systems typically truncate histories or rely on two-stage retrieve-then-rank pipelines, sacrificing long-term signals and breaking end-to-end optimization. While Stacked Target-to-History Cross Attention (STCA) enables end-to-end modeling up to the 10K regime, directly scaling it to ultra-long histories remains expensive in both training and serving because its per-layer cost grows linearly with the raw history length. We present SequenceO1, an end-to-end framework deployed at full traffic on Douyin at the 100K scale and designed to extend to million-scale histories. At the model level, we propose Sketch Attention (SA), which compresses an ultra-long history into a fixed-size, user-only sketch using learnable prototypes and prototype-wise normalization (each token distributes mass over prototypes). We then perform target-conditioned reasoning at two time scales: STCA over a recent 10K suffix for recency and STCA over the fixed-size sketch for ultra-long signals, followed by lightweight fusion. At the system level, a training-side local key–value cache reuses user-only sketches across repeated instances of the same user, eliminating n-dependent computation from model execution on cache hits; the same cacheable state is also reused across consecutive serving requests. We further improve efficiency with multi-request user-level batching in training and a fused FlashSA kernel for sketching under ragged batching. Together, these model and system optimizations make end-to-end 100K sequence modeling practical in production and provide a scalable path toward million-scale histories.