WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers

• URL: http://arxiv.org/abs/2606.06564v1
• Date: Thu, 04 Jun 2026 16:15:27 GMT
• Title: WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers
• Abstract summary: WAV v1 is a lightweight residual routing method for decoder-only Transformers.<n>It reduces loss relative to Block AttnRes from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8.
• Score: 1.2973070941583096
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Residual connections are central to training deep Transformers, but standard PreNorm residual streams aggregate sublayer updates with fixed unit weights. Recent Attention Residuals replace this fixed accumulation with content-dependent depth-wise routing, and Block Attention Residuals make the mechanism efficient by routing over block-level residual summaries. However, a single block summary stores only the low-frequency total residual displacement inside a block, discarding directional structure such as attention-vs-MLP imbalance and early-vs-late block dynamics. We propose WAV v1, a lightweight multi-resolution residual routing method for decoder-only Transformers. Instead of representing each block only by its accumulated residual sum, WAV v1 augments every block with two directional detail bases: a phase basis that contrasts attention and MLP updates, and a split basis that contrasts early and late sublayer updates. These bases are routed together with standard block summaries through the same depth-wise softmax mixer, while negative detail-source initialization and detached RMS matching stabilize training. On character-level TinyStories and Text8 language modeling, WAV v1 shows a clear depth-dependent benefit. Although it is not consistently beneficial at 12 layers, it becomes competitive at 24 layers and outperforms all baselines at 48 layers. At 48 layers, WAV v1 reduces validation loss relative to Block AttnRes from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8, with negligible additional parameters. These results suggest that directional residual details, not only block-level sums, are important for scaling residual routing in deeper Transformers.

Related papers

• SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs [18.494055578192015]
  We introduce SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter at each pruning site.<n>Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction.<n>Experiments on five model families, six layer-selection criteria, and seven zero-shot benchmarks show that SHIFT-LLM consistently recovers accuracy lost to depth pruning.
  arXiv  Detail & Related papers  (2026-08-25T19:01:58Z)
• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• Dual Attention Residuals [34.183227979638595]
  We propose Dual Attention Residuals, which brings multi-stream interaction into historical retrieval.<n>For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history.<n>DAR consistently improves validation loss over standard residual Transformers and Attention Residuals.
  arXiv  Detail & Related papers  (2026-07-21T05:42:55Z)
• Learning to Solve Generative ODEs Beyond the Linear Span [50.13853710831612]
  We propose SpanLift, a lightweight neural solver that augments scalar-coefficient updates with a spatial residual operator.<n>Across pixel-space diffusion, latent flow matching, and precipitation nowcasting, SpanLift achieves state-of-the-art few-step sampling.
  arXiv  Detail & Related papers  (2026-06-07T15:22:13Z)
• DeRes: Decoupling Residual Stability and Adaptivity for Scalable CTR Prediction [0.343054185715673]
  Transformer-based CTR models face a growing bottleneck at the residual connection.<n>Recent attention-based residual variants (AttnRes) address parts of this in language models.<n>We present DeRes, which routes each layer through two parallel paths.
  arXiv  Detail & Related papers  (2026-06-06T05:07:16Z)
• Replacement Learning: Training Neural Networks with Fewer Parameters [15.09968642484538]
  Replacement Learning (RepL) is a training-time paradigm that reduces full-depth redundancy by replacing selected blocks.<n>RepL preserves local contextual continuity while avoiding unnecessary full-layer.<n>We show that RepL reduces trainable parameters, GPU memory usage, and training time while matching or surpassing standard end-to-end training.
  arXiv  Detail & Related papers  (2026-05-19T08:34:31Z)
• Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers [0.2436681150766912]
  We show that networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation.<n>We propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement.
  arXiv  Detail & Related papers  (2026-05-07T12:53:45Z)
• Attention Residuals [38.59138244826294]
  Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights.<n>We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs.
  arXiv  Detail & Related papers  (2026-03-16T09:32:21Z)
• SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing [77.91660464664615]
  Diffusion Transformers (DiTs) have become a leading backbone for video generation, yet their quadratic attention cost remains a major bottleneck.<n>In this paper, we show that the missing contributions can be recovered without training.<n>We introduce SVG-EAR, a parameter-free linear compensation branch that uses the centroid to approximate skipped blocks and recover their contributions.
  arXiv  Detail & Related papers  (2026-03-09T22:15:31Z)
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs [58.640039233470766]
  We show that principled AR-to-block-diffusion adaptation is an effective and compute-efficient alternative to training DLMs from scratch.<n> NBDiff-7B (Base and Instruct) could inherit the long-context modeling and reasoning capabilities, and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-07T10:28:21Z)
• Arithmetic-Mean $μ$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets [9.94514344279733]
  Arithmetic-Mean $mu$P constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale.<n>We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies $etastar(L)propto L-3/2$; with zero padding, boundary effects are constant-level as $Ngg k$.
  arXiv  Detail & Related papers  (2025-10-05T19:22:50Z)
• The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training [51.84624027213658]
  We propose Blockwise Learning Rate (LR), a strategy that tailors the LR to each block's sharpness.<n>We demonstrate this acceleration across GPT-2 and LLaMA, with model sizes ranging from 0.12B to 2B.<n>We incorporate Blockwise LR into Adam-mini, a recently proposed memory-efficient variant of Adam, achieving a combined $2times$ speedup and $2times$ memory saving.
  arXiv  Detail & Related papers  (2025-02-26T10:06:37Z)
• Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization [65.33914980022303]
  Temporal action localization (TAL) requires long-form reasoning to predict actions of various durations and complex content. Most methods can only train on pre-extracted features without optimizing them for the localization problem. We propose a novel end-to-end method Re2TAL, which rewires pretrained video backbones for reversible TAL.
  arXiv  Detail & Related papers  (2022-11-25T12:17:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.