Dual Attention Residuals

• URL: http://arxiv.org/abs/2607.18730v1
• Date: Tue, 21 Jul 2026 05:42:55 GMT
• Title: Dual Attention Residuals
• Abstract summary: We propose Dual Attention Residuals, which brings multi-stream interaction into historical retrieval.<n>For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history.<n>DAR consistently improves validation loss over standard residual Transformers and Attention Residuals.
• Score: 34.183227979638595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Related papers

• DRIFT: Removing Diffusion Watermarks by Deflecting the Generative Trajectory [9.706332101916248]
  Diffusion watermarking embeds verifiable signals into the generative process.<n>We propose DRIFT, a black-box attack that combines partial forward diffusion with reverse resampling.<n>Across nine watermarks spanning three paradigms, DRIFT achieves 98-100% attack success and the best image quality.
  arXiv  Detail & Related papers  (2026-09-08T03:54:17Z)
• DUET -- Dual User Embedding Transformers for Offsite Conversion Prediction [8.972672149050627]
  Offsite conversion rate (OCVR) prediction is an important ranking problem in computational recommendation systems.<n>This task presents a modeling challenge: click signals are abundant and exhibit short temporal horizons, whereas conversion signals are inherently sparse, long-delayed, and frequently unattributed.<n>We propose DUET, a framework that explicitly partitions user behavioral data into two domain-coherent streams -- clicks and conversions.
  arXiv  Detail & Related papers  (2026-06-08T23:13:58Z)
• Action-Inspired Generative Models [0.0]
  Action-Inspired Generative Models (AGMs)<n>We introduce a dual-network generative framework motivated by the observation that existing bridge-matching methods assign uniform regression weight to every transition in the transport landscape.<n>We demonstrate that selectively penalising uninformative transport paths through the learned potential yields consistent improvements in generation quality across fidelity and coverage metrics.
  arXiv  Detail & Related papers  (2026-05-14T09:43:32Z)
• Beyond Reconstruction: Reconstruction-to-Vector Diffusion for Hyperspectral Anomaly Detection [9.078979356836193]
  Hyperspectral Anomaly Detection (HAD) excels at identifying sparse targets in complex scenes.<n>Existing models remain trapped in a scalar "reconstruction-as-endpoint" paradigm.<n>We propose Reconstruction-to-Diffusion (R2VD), which redefines reconstruction as a purification origin.<n>R2VD establishes a new state-of-the-art, delivering exceptional target detectability background.
  arXiv  Detail & Related papers  (2026-04-13T12:31:50Z)
• Robust Outlier Detection and Low-Latency Concept Drift Adaptation for Data Stream Regression: A Dual-Channel Architecture [9.977810035655805]
  Outlier detection and concept drift detection represent two challenges in data analysis.<n>We propose a novel robust regression framework for joint outlier and concept drift detection.<n>Our framework, enhanced by EWMAD-DT, exhibits superior detection performance even when point outliers and concept drifts coexist.
  arXiv  Detail & Related papers  (2025-12-13T11:17:47Z)
• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving [65.7087560656003]
  Generative diffusion models for end-to-end autonomous driving often suffer from mode collapse.<n>We propose DiffusionDriveV2, which leverages reinforcement learning to constrain low-quality modes and explore for superior trajectories.<n>This significantly enhances the overall output quality while preserving the inherent multimodality of its core Gaussian Mixture Model.
  arXiv  Detail & Related papers  (2025-12-08T17:29:52Z)
• Stable Video Infinity: Infinite-Length Video Generation with Error Recycling [76.91310169118408]
  We propose Stable Video Infinity (SVI) that is able to generate infinite-length videos with high temporal consistency, plausible scene transitions, and controllable streaming storylines.<n> SVI incorporates Error-Recycling Fine-Tuning, a new type of efficient training that recycles the Diffusion Transformer's self-generated errors into supervisory prompts.<n>We evaluate SVI on three benchmarks, including consistent, creative, and conditional settings, thoroughly verifying its versatility and state-of-the-art role.
  arXiv  Detail & Related papers  (2025-10-10T09:45:46Z)
• FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems [51.99765487172328]
  Posterior sampling for inverse problem solving can be effectively achieved using flows.<n>Flow-Driven Posterior Sampling (FlowDPS) outperforms state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2025-03-11T07:56:14Z)
• Amortized Posterior Sampling with Diffusion Prior Distillation [55.03585818289934]
  Amortized Posterior Sampling is a novel variational inference approach for efficient posterior sampling in inverse problems.<n>Our method trains a conditional flow model to minimize the divergence between the variational distribution and the posterior distribution implicitly defined by the diffusion model.<n>Unlike existing methods, our approach is unsupervised, requires no paired training data, and is applicable to both Euclidean and non-Euclidean domains.
  arXiv  Detail & Related papers  (2024-07-25T09:53:12Z)
• Haar Wavelet based Block Autoregressive Flows for Trajectories [129.37479472754083]
  Prediction of trajectories such as that of pedestrians is crucial to the performance of autonomous agents. We introduce a novel Haar wavelet based block autoregressive model leveraging split couplings. We illustrate the advantages of our approach for generating diverse and accurate trajectories on two real-world datasets.
  arXiv  Detail & Related papers  (2020-09-21T13:57:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.