Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders

• URL: http://arxiv.org/abs/2608.14021v1
• Date: Fri, 14 Aug 2026 07:09:58 GMT
• Title: Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders
• Abstract summary: We show that SASRec-style models exhibit highly localized last-item reliance.<n>We then find that residual addition sharply shifts the full-block representation toward same-position contributions.<n>Changing the residual strength induces a monotonic trade-off between structural mixing and last-item reliance.
• Score: 47.8060666024071
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based sequential recommenders with causal self-attention often rely heavily on the most recent interaction at inference time, but how this behavior is structurally expressed in the representation used for prediction remains unclear. We combine prediction-time diagnostics with norm-based analysis of the full attention block. First, we show that SASRec-style models exhibit highly localized last-item reliance. We then find that, although self-attention aggregates contextual information, residual addition sharply shifts the full-block representation toward same-position contributions, which we term residual dominance. To probe this interpretation, we use inference-time residual scaling as a controlled diagnostic intervention. Changing the residual strength induces a monotonic trade-off between structural mixing and last-item reliance, while reducing residual strength recovers a subset of final-position misses for which representations at non-final positions already rank the ground-truth item correctly. Our results provide a structural account linking extreme last-item reliance to residual dominance at inference time. The code is publicly available.

Related papers

• ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers [58.411900827911346]
  We show that the missing ingredient is the treatment of residual connections.<n>We introduce Residual-aware Layer-wise Relevance Propagation (ResLRP)<n>ResLRP is a simple extension of LRP whose propagation rules explicitly account for cancellations in residual branches.
  arXiv  Detail & Related papers  (2026-09-15T13:17:30Z)
• The Immutable Past: Formalizing State Mutability and Conflict Resolution in Mutable RAG [0.0]
  This paper formalizes the mechanics of State Mutability to prove that standard dense retrieval suffers from Asymptotic Recall Decay.<n>To resolve this, we introduce GC-Mem, a strict inference-time consistency protocol.
  arXiv  Detail & Related papers  (2026-09-13T15:28:07Z)
• When Interference Graphs Evolve: Doubly Robust Estimation of Dynamic Peer Effects [1.049126606580198]
  We introduce a controlled contrast framework that indexes potential outcomes by own treatment, temporally aggregated peer exposure, and a post-assignment evolution summary.<n>We develop the Dynamic Network Doubly Robust estimator, DynaNet-DR, which combines a temporally factorized propensity with normalized augmentation.
  arXiv  Detail & Related papers  (2026-08-27T14:32:40Z)
• Inference-Time Decision Calibration for Temporal Classification [0.607835026936425]
  Temporal classification errors can arise from how available evidence is converted into decisions.<n>This paper proposes a representation--calibration decomposition for temporal classification.
  arXiv  Detail & Related papers  (2026-06-14T22:02:48Z)
• AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE [51.994752158953084]
  AnchorMoE is an interpretable-by-construction classification framework.<n>It encodes multi-view representations of local patches and routes them to specialized experts.<n>Experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance.
  arXiv  Detail & Related papers  (2026-06-02T13:30:54Z)
• The Need for an External Observer Formalizing the Sufficiency Gap: A Mathematical Extension of Mixture Identifiability and Contextual Grounding in Sequence Models [0.0]
  We construct a binary mixed-regime process with one deterministic textual regime and one random regime governed by an unobserved latent state.<n>The resulting entropy difference is not an ordinary optimization error; it is a sufficiency gap caused by marginalization over an unobserved state.<n>A corrective signal reverses the posterior odds induced by the textual history exactly when its fidelity exceeds the text-only posterior weight assigned to the misleading regime.
  arXiv  Detail & Related papers  (2026-05-26T08:53:11Z)
• Physics-Guided Self-Supervised Statistical Residual Learning for Sonar Despeckling with Improved Generalization [1.9371675344367494]
  This letter introduces a physics-informed self-supervised framework for sonar image despeckling that reformulates despeckling as residual consistency in the homomorphic log domain.<n>A variance-targeted statistical loss combined with edge-aware structural regularization and median-guided curriculum stabilization enables effective speckle suppression with preserved structural fidelity.
  arXiv  Detail & Related papers  (2026-05-23T20:07:49Z)
• When Attention Collapses: Residual Evidence Modeling for Compositional Inference [0.0]
  Under additive superposition, multiple slots converge to the same dominant component while weaker ones remain unrepresented.<n>We introduce residual evidence modeling, instantiated via evidence depletion, a minimal modification combining multiplicative depletion with an attention bias.<n>Across synthetic benchmarks and real-world audio mixtures, evidence depletion reduces slot collapse by up to an order of magnitude.
  arXiv  Detail & Related papers  (2026-05-04T08:18:02Z)
• Explicit Dropout: Deterministic Regularization for Transformer Architectures [55.09895958546215]
  Dropout is a widely used regularization technique in deep learning, but its effects are typically realized through masking.<n>We propose a deterministic formulation that expresses dropout as an additive regularizer directly incorporated into the training loss.
  arXiv  Detail & Related papers  (2026-04-22T12:45:51Z)
• APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards [61.52322047892064]
  Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs)<n>We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process.<n>We propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST.
  arXiv  Detail & Related papers  (2026-01-31T14:53:20Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.