SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching

• URL: http://arxiv.org/abs/2609.08443v1
• Date: Tue, 08 Sep 2026 08:49:44 GMT
• Title: SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching
• Abstract summary: SequenceO1 is an end-to-end framework for ultra-long user behavior sequence modeling.<n>It is deployed at full traffic on Douyin with histories of up to 100K interactions.<n>Results show consistent offline and online gains, while sketch retains most of the benefit of directly scaling end-to-end sequence ranking to 100K.
• Score: 17.335538522124555
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modeling long-term user behavior is central to sequential recommendation and billion-scale industrial recommender systems, yet production ranking models operate under strict latency, memory, communication, and training-throughput constraints. At the 100K scale, the challenge extends beyond attention complexity: raw sequence features must be stored, transferred, and repeatedly processed during training and online serving. Existing approaches based on history truncation, multi-stage behavior retrieval, compressed lifelong histories, or train-short/infer-long extrapolation either weaken end-to-end optimization or retain substantial length-dependent cost. We present SequenceO1, an end-to-end framework for ultra-long user behavior sequence modeling, deployed at full traffic on Douyin with histories of up to 100K interactions. SequenceO1 follows a compress-then-reason design. Its Sketch Attention (SA) uses learnable prototypes and prototype-wise normalization to compress the raw history into a fixed-size, target-agnostic user representation. Target-conditioned Stacked Target-to-History Cross Attention (STCA) then models complementary time scales: a recent 10K suffix for short-term interests and the compact sketch for long-term preferences. To make training and inference practical, SequenceO1 combines low-rank user representation caching, multi-request user-level batching, pipeline lift, and a fused FlashSA kernel to amortize feature storage, communication, and computation across targets, training instances, and consecutive requests. Production experiments show consistent offline and online gains, while the compact cached sketch retains most of the benefit of directly scaling end-to-end sequence ranking to 100K. These results provide a practical model-system approach to efficient attention, sequence compression, and scalable long-sequence and long-context recommendation systems.

Related papers

• Rethinking Convolutional Networks for Attribute-Aware Sequential Recommendation [6.846413131554734]
  Attribute-aware sequential recommendation entails predicting the next item a user will interact with based on a chronologically ordered history of past interactions.<n>Existing methods typically leverage self-attention mechanisms to aggregate the entire sequence into a unified representation.<n>We propose ConvRec, which employs convolutional layers in a hierarchical, down-scaled fashion to generate compact, yet expressive sequence representations.
  arXiv  Detail & Related papers  (2026-05-06T10:18:43Z)
• Action-Aware Generative Sequence Modeling for Short Video Recommendation [50.74467504063892]
  Short videos often contain diverse segments, and users may not hold the same attitude toward all of them.<n>Traditional binary-classification recommendation models, which treat a video as a single holistic entity, face limitations in accurately capturing such nuanced preferences.<n>We propose a novel modeling paradigm: Action-Aware Generative Sequence Network (A2Gen), which refines user actions along the temporal dimension.
  arXiv  Detail & Related papers  (2026-04-28T16:41:04Z)
• GEMs: Breaking the Long-Sequence Barrier in Generative Recommendation with a Multi-Stream Decoder [54.64137490632567]
  We propose a novel and unified framework designed to capture users' sequences from long-term history.<n>Generative Multi-streamers ( GEMs) break user sequences into three streams.<n>Extensive experiments on large-scale industrial datasets demonstrate that GEMs significantly outperforms state-the-art methods in recommendation accuracy.
  arXiv  Detail & Related papers  (2026-02-14T06:42:56Z)
• Length-Adaptive Interest Network for Balancing Long and Short Sequence Modeling in CTR Prediction [50.094751096858204]
  LAIN is a plug-and-play framework that incorporates sequence length as a conditioning signal to balance long- and short-sequence modeling.<n>Our work offers a general, efficient, and deployable solution to mitigate length-induced bias in sequential recommendation.
  arXiv  Detail & Related papers  (2026-01-27T03:14:20Z)
• Make It Long, Keep It Fast: End-to-End 10k-Sequence Modeling at Billion Scale on Douyin [21.0248704845397]
  Short-video recommenders such as Douyin must exploit extremely long user histories without breaking latency or cost budgets.<n>We present an end-to-end system that scales long-length modeling to 10k histories in production histories.
  arXiv  Detail & Related papers  (2025-11-08T17:22:54Z)
• OmniSAT: Compact Action Token, Faster Auto Regression [70.70037017501357]
  We introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation.<n>The resulting discrete tokenization shortens the training sequence by 6.8$times$, and lowers the target entropy.
  arXiv  Detail & Related papers  (2025-10-08T03:55:24Z)
• Long-Sequence Recommendation Models Need Decoupled Embeddings [49.410906935283585]
  We identify and characterize a neglected deficiency in existing long-sequence recommendation models.<n>A single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes.<n>We propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are learned separately to fully decouple attention and representation.
  arXiv  Detail & Related papers  (2024-10-03T15:45:15Z)
• TWIN V2: Scaling Ultra-Long User Behavior Sequence Modeling for Enhanced CTR Prediction at Kuaishou [28.809014888174932]
  We introduce TWIN-V2, an enhancement of SIM, where a divide-and-conquer approach is applied to compress life-cycle behaviors and uncover more accurate and diverse user interests. Under an efficient deployment framework, TWIN-V2 has been successfully deployed to the primary traffic that serves hundreds of millions of daily active users at Kuaishou.
  arXiv  Detail & Related papers  (2024-07-23T10:00:45Z)
• Efficient User Sequence Learning for Online Services via Compressed Graph Neural Networks [18.068737439570402]
  Graph Neural Networks (GNNs) have been extensively applied to model sequence relationships, and extract information from similar sequences. We propose a novel unified framework called ECSeq, to introduce graph compression techniques into relation modeling for user sequence representation learning.
  arXiv  Detail & Related papers  (2024-06-05T06:22:11Z)
• Sparse Attentive Memory Network for Click-through Rate Prediction with Long Sequences [10.233015715433602]
  We propose a Sparse Attentive Memory network for long sequential user behavior modeling. SAM supports efficient training and real-time inference for user behavior sequences with lengths on the scale of thousands. SAM is successfully deployed on one of the largest international E-commerce platforms.
  arXiv  Detail & Related papers  (2022-08-08T10:11:46Z)
• Dynamic Memory based Attention Network for Sequential Recommendation [79.5901228623551]
  We propose a novel long sequential recommendation model called Dynamic Memory-based Attention Network (DMAN) It segments the overall long behavior sequence into a series of sub-sequences, then trains the model and maintains a set of memory blocks to preserve long-term interests of users. Based on the dynamic memory, the user's short-term and long-term interests can be explicitly extracted and combined for efficient joint recommendation.
  arXiv  Detail & Related papers  (2021-02-18T11:08:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.