From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs

• URL: http://arxiv.org/abs/2609.01240v1
• Date: Tue, 01 Sep 2026 13:40:36 GMT
• Title: From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs
• Abstract summary: ReST is a recommendation-native Transformer scaling framework.<n>For signal quality, it introduces a sequence encoder with dual-gated attention.<n>For asymmetry, it factorizes ranking into a heavy reusable encoder and decoder.
• Score: 20.42858135695384
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling Transformers has driven large gains in language modeling, but transplanting this to behavior-sequence modeling in production ranking is challenging: recommendation differs in signal quality, where behavior sequences are noisy, temporally irregular, and sparsely supervised, and in computation asymmetry, where each request scores many candidates against one shared user history under tight latency budgets. We propose ReST, a recommendation-native Transformer scaling framework. For signal quality, it introduces a sequence encoder with dual-gated attention, rotary positional and temporal embedding, stabilized residual normalization, and training-only auxiliary objectives. For computation asymmetry, it factorizes ranking into a heavy reusable encoder and a lightweight cross decoder with projection-free KV attention and token-specific parameterization, coupling user-level shared-prefix training with shared-prefix serving for compute-once, decode-many-times ranking. Across industrial and public benchmarks, ReST achieves higher accuracy and scales more consistently along sequence length, depth, and width, where LLM-style Transformer blocks saturate. A one-week online A/B test on a production advertising platform improves online AUC by 1.31% and lifts a core revenue metric by 11.93% within a 50 ms P99 budget; ReST has since been fully deployed in production, showing that behavior-sequence scaling remains a promising, under-exploited axis for production ranking.

Related papers

• TransRetrieval: Scaling Up Transformer-Based Retrieval for Industrial Recommendation [19.45139517853295]
  We present TransRetrieval, a Transformer-based retrieval framework that scales with both computational budget and cross-domain data.<n>On a 40-billion-interaction industrial dataset, scaling compute from 0.1 to 2 MFLOPs per target yields +19.3/+22.2 pt Recall@2000.<n>In online A/B tests, TransRetrieval lifts platform revenue by 2.53% under the same end-to-end latency constraint as the production baseline.
  arXiv  Detail & Related papers  (2026-08-26T08:35:16Z)
• TransX: Scaling Transformer-based Recommendation via Behavioral and Serving Stream Crossings [4.750812376477589]
  We propose TransX, a production-oriented encoder-decoder architecture that reformulates recommendation as a sequence-to-sequence action problem.<n>To enable low-latency, high-QPS deployment, TransX is co-designed with an amortized serving strategy that combines incremental behavior encoding with per-request key-value caching.
  arXiv  Detail & Related papers  (2026-07-31T01:42:57Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• CoRe: A Continuously Reward-Finetuned LLM Query Rewriter for Multi-Stage Context-Aware Relevance in Web-Scale Video Search [16.32525191514382]
  We present CoRe, a system that redeployed weekly for over five months in a major short-video search engine.<n>Our reward uses the deployed multimodal relevance model as its source and a multiplicative ratio form mirroring the production fusion algebra.<n>A semi-online Mixed Preference Optimization loop makes this reward affordable at multi-million-instance weekly scale.
  arXiv  Detail & Related papers  (2026-06-12T05:19:40Z)
• DUET -- Dual User Embedding Transformers for Offsite Conversion Prediction [8.972672149050627]
  Offsite conversion rate (OCVR) prediction is an important ranking problem in computational recommendation systems.<n>This task presents a modeling challenge: click signals are abundant and exhibit short temporal horizons, whereas conversion signals are inherently sparse, long-delayed, and frequently unattributed.<n>We propose DUET, a framework that explicitly partitions user behavioral data into two domain-coherent streams -- clicks and conversions.
  arXiv  Detail & Related papers  (2026-06-08T23:13:58Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• COMET-SG1: Lightweight Autoregressive Regressor for Edge and Embedded AI [0.0]
  COMET-SG1 is a stability-oriented autoregressive regression model designed for time-series prediction on edge and embedded AI systems.<n>It operates through linear behavior-space encoding, memory-anchored transition estimation, and deterministic state updates.
  arXiv  Detail & Related papers  (2026-01-28T16:59:56Z)
• GSPN-2: Efficient Parallel Sequence Modeling [101.33780567131716]
  Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme.<n>GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications.
  arXiv  Detail & Related papers  (2025-11-28T07:26:45Z)
• OmniSAT: Compact Action Token, Faster Auto Regression [70.70037017501357]
  We introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation.<n>The resulting discrete tokenization shortens the training sequence by 6.8$times$, and lowers the target entropy.
  arXiv  Detail & Related papers  (2025-10-08T03:55:24Z)
• Continual Low-Rank Scaled Dot-product Attention [67.11704350478475]
  We introduce a new formulation of the Scaled-product Attention based on the Nystr"om approximation that is suitable for Continual Inference.<n>In experiments on Online Audio Classification and Online Action Detection tasks, the proposed Continual Scaled Dot-product Attention can lower the number of operations by up to three orders of magnitude.
  arXiv  Detail & Related papers  (2024-12-04T11:05:01Z)
• RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration [73.69415797389195]
  We propose an end-to-end transformer network (RegFormer) for large-scale point cloud alignment. Specifically, a projection-aware hierarchical transformer is proposed to capture long-range dependencies and filter outliers. Our transformer has linear complexity, which guarantees high efficiency even for large-scale scenes.
  arXiv  Detail & Related papers  (2023-03-22T08:47:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.