LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training

• URL: http://arxiv.org/abs/2609.14978v1
• Date: Mon, 14 Sep 2026 03:31:59 GMT
• Title: LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training
• Abstract summary: We propose a scaling Transformer for Industrial Recommendation via Transferable Generative Pre-training, termed LazFormer.<n>Specifically, we first present a generative pre-training module to autoregressively generate sequential features.<n>We then propose a transferable residual adapter that injects additional ranking-specific features into ranking in a residual manner.
• Score: 15.987387096597496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers have shown promising performance in LLMs due to their outstanding scalability, several studies have investigated the scalability of Transformers for industrial recommendation. They typically rely on a single ranking model to optimize both sparse and dense parameters from scratch, resulting in substantial computational resource consumption and slow convergence. Fortunately, the pre-training models offer an effective solution to the above issues by providing favorable initialization of both sparse and dense parameters for the subsequent ranking. However, they still face two major limitations: (1) Since the input features used in pre-training and ranking are usually inconsistent, directly transferring dense parameters from pre-training to ranking may lead to negative transfer. (2) Multi-epoch training during the ranking process may result in the overfitting of sparse parameters, while freezing the sparse parameters limits their adaptability to the ranking objectives. To this end, we propose a Scaling Transformer for Industrial Recommendation via Transferable Generative Pre-training, termed LazFormer. Specifically, we first present a generative pre-training module to autoregressively generate sequential features, providing favorable initialization of both sparse and dense parameters for the subsequent ranking. To solve the negative transfer of dense parameters, we propose a transferable residual adapter that injects additional ranking-specific features into ranking in a residual manner. Moreover, a request-aware ranking module integrates long-sequence compression, hybrid sparse attention, and a request-aware paradigm to efficiently model users' long sequences. Besides, we further propose an asymmetric multi-epoch training strategy that resets sparse parameters while continuously accumulating dense parameters across epochs, alleviating the overfitting of sparse parameters.

Related papers

• TS-Memory: Plug-and-Play Memory for Time Series Foundation Models [63.21390142212087]
  Time Series Foundation Models (TSFMs) achieve strong zero-shot forecasting through large-scale pre-training.<n>Existing solutions face a trade-off: Parametric Adaptation can cause catastrophic forgetting, while Non-Parametric Retrieval improves forecasts but incurs high latency due to datastore search.<n>We propose Parametric Memory Distillation and implement it as TS-Memory, a lightweight memory adapter that augments frozen TSFMs.
  arXiv  Detail & Related papers  (2026-02-12T04:16:19Z)
• High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>We present textbfSMoA, a high-rank textbfStructured textbfMOdulation textbfAdapter that uses fewer trainable parameters while maintaining a higher rank.
  arXiv  Detail & Related papers  (2026-01-12T13:06:17Z)
• Compressing Sine-Activated Low-Rank Adapters through Post-Training Quantization [25.441086332799348]
  Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning.<n>We extend the sinusoidal transformation framework to quantized LoRA adapters.
  arXiv  Detail & Related papers  (2025-05-28T02:15:15Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• LoRTA: Low Rank Tensor Adaptation of Large Language Models [70.32218116940393]
  Low Rank Adaptation (LoRA) is a popular Efficient Fine Tuning (PEFT) method.<n>We propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation.<n>Our method can achieve a reduction in the number of parameters while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-10-05T06:59:50Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Few-Shot Class Incremental Learning via Robust Transformer Approach [16.590193619691416]
  Few-Shot Class-Incremental Learning presents an extension of the Class Incremental Learning problem where a model is faced with the problem of data scarcity. This problem remains an open problem because all recent works are built upon the convolutional neural networks performing sub-optimally. Our paper presents Robust Transformer Approach built upon the Compact Convolution Transformer.
  arXiv  Detail & Related papers  (2024-05-08T03:35:52Z)
• PVP: Pre-trained Visual Parameter-Efficient Tuning [29.05396521860764]
  Large-scale pre-trained transformers have demonstrated remarkable success in various computer vision tasks. It is still highly challenging to fully fine-tune these models for downstream tasks due to their high computational and storage costs. We propose a Pre-trained Visual. efficient (PVP) Tuning framework, which pre-trains the parameter-efficient tuning modules first and then leverages the pre-trained modules.
  arXiv  Detail & Related papers  (2023-04-26T15:55:29Z)
• RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration [73.69415797389195]
  We propose an end-to-end transformer network (RegFormer) for large-scale point cloud alignment. Specifically, a projection-aware hierarchical transformer is proposed to capture long-range dependencies and filter outliers. Our transformer has linear complexity, which guarantees high efficiency even for large-scale scenes.
  arXiv  Detail & Related papers  (2023-03-22T08:47:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.