Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

• URL: http://arxiv.org/abs/2603.10369v1
• Date: Wed, 11 Mar 2026 03:30:49 GMT
• Title: Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems
• Abstract summary: We introduce Attention-based Late Fusion for Actions (AttnLFA) and Attention-based Mixed Value Pooling (AttnMVP)<n>These models explicitly encode the $i_n rightarrow a_n$ causal dependency while preserving the expressive power of Transformer-based sequence modeling.<n>We evaluate our framework on large-scale product recommendation data from a major social network.
• Score: 0.14930644306169805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative Recommender Systems (GR) increasingly model user behavior as a sequence generation task by interleaving item and action tokens. While effective, this formulation introduces significant structural and computational inefficiencies: it doubles sequence length, incurs quadratic overhead, and relies on implicit attention to recover the causal relationship between an item and its associated action. Furthermore, interleaving heterogeneous tokens forces the Transformer to disentangle semantically incompatible signals, leading to increased attention noise and reduced representation efficiency.In this work, we propose a principled reformulation of generative recommendation that aligns sequence modeling with underlying causal structures and attention theory. We demonstrate that current interleaving mechanisms act as inefficient proxies for similarity-weighted action pooling. To address this, we introduce two novel architectures that eliminate interleaved dependencies to reduce sequence complexity by 50%: Attention-based Late Fusion for Actions (AttnLFA) and Attention-based Mixed Value Pooling (AttnMVP). These models explicitly encode the $i_n \rightarrow a_n$ causal dependency while preserving the expressive power of Transformer-based sequence modeling.We evaluate our framework on large-scale product recommendation data from a major social network. Experimental results show that AttnLFA and AttnMVP consistently outperform interleaved baselines, achieving evaluation loss improvements of 0.29% and 0.80%, and significant gains in Normalized Entropy (NE). Crucially, these performance gains are accompanied by training time reductions of 23% and 12%, respectively. Our findings suggest that explicitly modeling item-action causality provides a superior design paradigm for scalable and efficient generative ranking.

Related papers

• Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling [61.28599393707943]
  Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps.<n>We propose a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment.<n> Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03% of dense understanding performance with a 1.93times$ end-to-end inference speedup.
  arXiv  Detail & Related papers  (2026-08-29T14:27:55Z)
• Beyond the Flat Sequence: Hierarchical and Preference-Aware Generative Recommendations [35.58864660038236]
  We propose a novel framework named HPGR (Hierarchical and Preference-aware Generative Recommender)<n>First, a structure-aware pre-training stage employs a session-based Masked Item Modeling objective to learn a hierarchically-informed and semantically rich item representation space.<n>Second, a preference-aware fine-tuning stage leverages these powerful representations to implement a Preference-Guided Sparse Attention mechanism.
  arXiv  Detail & Related papers  (2026-03-01T08:15:34Z)
• MixFormer: Co-Scaling Up Dense and Sequence in Industrial Recommenders [11.566232697512879]
  MixFormer is a unified Transformer-style architecture tailored for recommender systems.<n>It jointly models sequential behaviors and feature interactions within a single backbone.<n>Experiments on large-scale industrial datasets demonstrate that MixFormer consistently exhibits superior accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-02-15T11:53:30Z)
• PRISM: Parallel Residual Iterative Sequence Model [52.26239951489612]
  We propose PRISM (Parallel Residual Iterative Sequence Model) to resolve this tension.<n>PRISM introduces a solver-inspired inductive bias that captures key structural properties of multi-step refinement in a parallelizable form.<n>We prove that this formulation achieves Rank-$L$ accumulation, structurally expanding the update manifold beyond the single-step Rank-$1$ bottleneck.
  arXiv  Detail & Related papers  (2026-02-11T12:39:41Z)
• Repulsor: Accelerating Generative Modeling with a Contrastive Memory Bank [65.00301565190824]
  mname is a plug-and-play training framework that requires no external encoders.<n>mname achieves a state-of-the-art FID of textbf2.40 within 400k steps, significantly outperforming comparable methods.
  arXiv  Detail & Related papers  (2025-12-09T14:39:26Z)
• HoMer: Addressing Heterogeneities by Modeling Sequential and Set-wise Contexts for CTR Prediction [10.779868699316829]
  We propose HoMer, a Homogeneous-Oriented TransforMer for modeling sequential and set-wise contexts.<n>HoMer scales up and outperforms our industrial baseline by 0.0099 in the AUC metric, and enhances online business metrics like CTR/RPM by 1.99%/2.46%.
  arXiv  Detail & Related papers  (2025-10-13T07:47:03Z)
• Surrogate Modeling via Factorization Machine and Ising Model with Enhanced Higher-Order Interaction Learning [4.327676717016638]
  We propose an enhanced surrogate model that incorporates additional slack variables into both the factorization machine and its associated Ising representation.<n>We apply the proposed method to the task of predicting drug combination effects.
  arXiv  Detail & Related papers  (2025-07-02T06:10:49Z)
• Killing Two Birds with One Stone: Unifying Retrieval and Ranking with a Single Generative Recommendation Model [71.45491434257106]
  Unified Generative Recommendation Framework (UniGRF) is a novel approach that integrates retrieval and ranking into a single generative model.<n>To enhance inter-stage collaboration, UniGRF introduces a ranking-driven enhancer module.<n>UniGRF significantly outperforms existing models on benchmark datasets.
  arXiv  Detail & Related papers  (2025-04-23T06:43:54Z)
• Decoupled Sequence and Structure Generation for Realistic Antibody Design [45.72237864940556]
  A dominant paradigm is to train a model to jointly generate the antibody sequence and the structure as a candidate.<n>We propose an antibody sequence-structure decoupling (ASSD) framework, which separates sequence generation and structure prediction.<n>ASSD shows improved performance in various antibody design experiments, while the composition-based objective successfully mitigates token repetition of non-autoregressive models.
  arXiv  Detail & Related papers  (2024-02-08T13:02:05Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• Causal Incremental Graph Convolution for Recommender System Retraining [89.25922726558875]
  Real-world recommender system needs to be regularly retrained to keep with the new data. In this work, we consider how to efficiently retrain graph convolution network (GCN) based recommender models.
  arXiv  Detail & Related papers  (2021-08-16T04:20:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.