DeRes: Decoupling Residual Stability and Adaptivity for Scalable CTR Prediction

• URL: http://arxiv.org/abs/2606.07980v1
• Date: Sat, 06 Jun 2026 05:07:16 GMT
• Title: DeRes: Decoupling Residual Stability and Adaptivity for Scalable CTR Prediction
• Abstract summary: Transformer-based CTR models face a growing bottleneck at the residual connection.<n>Recent attention-based residual variants (AttnRes) address parts of this in language models.<n>We present DeRes, which routes each layer through two parallel paths.
• Score: 0.343054185715673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based CTR models face a growing bottleneck at the residual connection: under Pre-Norm, early user-interest signals are diluted layer by layer; the identity skip cannot forget stale interests; and each layer sees only its immediate predecessor, losing long-range cross-layer dependencies. Recent attention-based residual variants (AttnRes) address parts of this in language models, but drop the protective identity skip and have not been tried in recommendation. Drawing on Dual Path Networks (DPN) and the HORNN view of residuals, we present DeRes, which routes each layer through two parallel paths -- an Identity residual path that preserves first-order feature reuse and gradient flow, and a Block Attention Residual path that attends over compressed outputs of all earlier blocks for high-order recall. A vector-wise gate decides, per hidden dimension, the weight given to each path. We further propose Pointwise AttnRes, replacing the Softmax in the cross-layer attention with SiLU so that multiple past blocks can be activated simultaneously and irrelevant ones receive negative (forgetting) weights -- better aligned with CTR's parallel multi-interest patterns. On a large-scale industrial dataset (331M interactions from a major social-media platform), Criteo (45M), and Avazu (40M), DeRes outperforms twelve baselines including OneTrans, TokenMixer-Large, UniMixer, mHC, and AttnRes, achieving up to +0.32% AUC at under 5% extra FLOPs. Beyond a single operating point, DeRes fits a markedly steeper compute-AUC scaling law (gamma=0.118 vs. 0.071 for OneTrans, a 1.66x gap), so an 8-layer DeRes matches a 16-layer OneTrans -- about 2x compute saving at equivalent AUC. Ablations confirm that the dual-path design outperforms either single path, Identity beats learnable residuals, and SiLU beats Softmax.

Related papers

• MergeOver: Post-Training Token Merging for Recursive Vision Transformers [1.6106006867887395]
  MergeOver integrates Token Merging (ToMe) into the Sliced Recursive Transformer (SReT)<n> Benchmarked on ImageNet-1K, our selected configuration reduces top-1 accuracy by 1.47 percentage points.<n>On the GPU, it reduces peak activation memory by 37.3% and 38.4% at batch sizes 1 and 16, while throughput decreases by 21.7% at batch size 1 but increases by 21.7% at batch size 16.
  arXiv  Detail & Related papers  (2026-08-13T12:15:10Z)
• Coupled Continuous-Discrete Generation for Scene Text Image Super-Resolution [51.31474187379532]
  Scene text image super-resolution (STISR) aims to recover visually plausible appearance while preserving character semantics from degraded inputs.<n>We present DualTSR, a unified framework that formulates STISR as coupled continuous-discrete generation.<n> Conditional flow matching restores continuous image latents, while absorbing-state discrete diffusion reconstructs text tokens.
  arXiv  Detail & Related papers  (2026-08-05T06:57:30Z)
• TL++: Accuracy and Privacy Preserving Traversal Learning for Distributed Intelligent Systems [0.0]
  We present TL++, a two-mode-learning framework that constructs virtual batches across nodes to recover centralized mini-batch gradient behavior.<n>Secure mode secret-shares each cut-layer activation and gradient between an orchestrator and a non-colluding helper.<n>On CIFAR-10, TL++ base cut 1 and exact secure cut 3 achieve accuracies of 91.41% (SD 0.19) and 90.93% (SD 0.179), respectively, exceeding the strongest measured non-TL++ baseline by more than 12 percentage points.
  arXiv  Detail & Related papers  (2026-06-24T09:34:27Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers [1.2973070941583096]
  WAV v1 is a lightweight residual routing method for decoder-only Transformers.<n>It reduces loss relative to Block AttnRes from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8.
  arXiv  Detail & Related papers  (2026-06-04T16:15:27Z)
• Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders [123.58723804218151]
  Vision foundation models are bottlenecked by the quadratic cost of self-attention.<n>We present C-GSPN based on 2D spatial propagation propagation.
  arXiv  Detail & Related papers  (2026-05-30T14:29:43Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Attention Residuals [38.59138244826294]
  Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights.<n>We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs.
  arXiv  Detail & Related papers  (2026-03-16T09:32:21Z)
• FMS$^2$: Unified Flow Matching for Segmentation and Synthesis of Thin Structures [1.3111058225843968]
  We propose FMS$2$, a flow-matching framework with two modules.<n>SegFlow is a 2.96M- parameter segmentation model built on a standard encoder-decoder backbone.<n> SynFlow is a mask-conditioned mask $rightarrow$ image generator that produces pixel-aligned synthetic image-mask pairs.
  arXiv  Detail & Related papers  (2026-03-14T00:04:25Z)
• Unleashing Degradation-Carrying Features in Symmetric U-Net: Simpler and Stronger Baselines for All-in-One Image Restoration [52.82397287366076]
  All-in-one image restoration aims to handle diverse degradations (e.g., noise, blur, adverse weather) within a unified framework.<n>In this work, we reveal a critical insight: well-crafted feature extraction inherently encodes degradation-carrying information.<n>Our symmetric design preserves intrinsic degradation signals robustly, rendering simple additive fusion in skip connections.
  arXiv  Detail & Related papers  (2025-12-11T12:20:31Z)
• GSPN-2: Efficient Parallel Sequence Modeling [101.33780567131716]
  Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme.<n>GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications.
  arXiv  Detail & Related papers  (2025-11-28T07:26:45Z)
• Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss [59.835032408496545]
  We propose a tile-based strategy that partitions the contrastive loss calculation into arbitrary small blocks. We also introduce a multi-level tiling strategy to leverage the hierarchical structure of distributed systems. Compared to SOTA memory-efficient solutions, it achieves a two-order-of-magnitude reduction in memory while maintaining comparable speed.
  arXiv  Detail & Related papers  (2024-10-22T17:59:30Z)
• RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network [3.54359747576165]
  RevSilo is the first reversible multi-scale feature fusion module. We create RevBiFPN, a fully reversible bidirectional feature pyramid network. RevBiFPN provides up to a 2.5% boost in AP over HRNet using fewer MACs and a 2.4x reduction in training-time memory.
  arXiv  Detail & Related papers  (2022-06-28T15:48:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.