Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth

• URL: http://arxiv.org/abs/2608.01075v1
• Date: Sun, 02 Aug 2026 08:19:39 GMT
• Title: Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
• Abstract summary: We introduce Role-Decoupled Attention Residuals (RD-AttnRes)<n>RD-AttnRes shares one depth route between queries and keys while learning an independent value route over the same residual sources.<n>We evaluate RD-AttnRes using a frozen, paired pretraining protocol on FineWeb-Edu.
• Score: 1.2973070941583096
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Depth-routing residual architectures allow Transformer layers to retrieve earlier representations instead of inheriting only the immediately preceding state. Existing Block Attention Residuals, however, use a single content-dependent depth mixture to construct the inputs to queries, keys, and values. This design couples two functionally different decisions: queries and keys determine where attention matches, whereas values determine what content is retrieved. We therefore ask whether matching and content retrieval should be forced to read from the same depth. We introduce Role-Decoupled Attention Residuals (RD-AttnRes), a minimal extension that shares one depth route between queries and keys while learning an independent value route over the same residual sources. Tying the two routing queries exactly recovers the parent architecture, while decoupling them adds only one model-width vector per layer and introduces no additional token-to-token attention operation. We evaluate RD-AttnRes using a frozen, paired pretraining protocol on FineWeb-Edu with five matched seeds for both 120M- and 343M-parameter models and a 2.0B-token training budget. RD-AttnRes improves validation negative log-likelihood in all 10 matched comparisons. The mean reductions are 0.0301 and 0.0247, corresponding to perplexity reductions of 2.97 percent and 2.43 percent at 120M and 343M parameters, respectively. Early-budget controls indicate that neither the additional parameter count, duplicated routing execution, nor a fixed value route reproduces the improvement. Routing diagnostics further reveal persistent divergence between the query-key and value depth distributions. These results suggest that, within the evaluated training regime, attention matching and content retrieval benefit from distinct reads over the residual hierarchy.

Related papers

• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification [71.9783246097687]
  SEER hides candidate relations during pair localization, constructs a query-specific view with explicit subject/object roles, and retains the full image and sparse box geometry as complementary evidence.<n>For relation-choice protocols with exact inverse support, an optional refinement swaps the entity roles and changes the forward decision only when exactly one visual state obeys the corresponding inverse relation.<n>The unchanged protocol yields +4.35 to +11.79 on all 2,434 filtered EmbSpatial pair-relation questions across three models.
  arXiv  Detail & Related papers  (2026-08-04T13:16:15Z)
• Interdomain Attention: Beyond Token-Level Key-Value Memory [15.708551010135407]
  We propose Interdomain Attention, which integrates an SSM into an attention module through kernel methods.<n>An attention kernel is approximated by a finite feature map, the resulting key features and values are projected onto a shared set of basis functions.<n>Each query attends to the compressed coefficients through its own feature map, recovering query-conditioned attention over a fixed-size state.
  arXiv  Detail & Related papers  (2026-05-23T01:18:00Z)
• Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning [14.508049757681862]
  LLM-powered coding agents spend the majority of their token budget reading repository files.<n>We show that this formulation creates a modeling bottleneck.<n>We propose LaMR, a structured pruning framework that decomposes code relevance into two interpretable quality dimensions.
  arXiv  Detail & Related papers  (2026-05-14T18:30:10Z)
• TRU: Targeted Reverse Update for Efficient Multimodal Recommendation Unlearning [10.314982171290835]
  Multimodal recommendation systems (MRS) jointly model user-item interaction graphs and rich item content, but this tight coupling makes user data difficult to remove once learned.<n> Approximate machine unlearning offers an efficient alternative to full retraining, yet existing methods for MRS mainly rely on a largely uniform reverse update across the model.<n>We show that this assumption is fundamentally mismatched to modern MRS: deleted-data influence is not uniformly distributed, but concentrated unevenly across textitranking behavior, textitmodality branches, and textitnetwork layers.
  arXiv  Detail & Related papers  (2026-04-02T15:45:03Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• Route-DETR: Pairwise Query Routing in Transformers for Object Detection [11.46025964297103]
  Detection Transformer (DETR) offers an end-to-end solution for object detection.<n>DETR suffers from inefficient query competition where multiple queries converge to similar positions.<n>We present Route-DETR, which addresses these issues through adaptive pairwise routing in decoder self-attention layers.
  arXiv  Detail & Related papers  (2025-12-15T20:26:58Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• Hybrid Deep Searcher: Integrating Parallel and Sequential Search Reasoning [57.78245296980122]
  We introduce HDS-QA (Hybrid Deep Search QA), a dataset automatically generated from Natural Questions.<n>It comprises hybrid-hop questions that combine parallelizable independent subqueries (executable simultaneously) and sequentially dependent subqueries (requiring step-by-step resolution)<n>We name the model HybridDeepSearcher, which outperforms state-of-the-art baselines across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-08-26T15:15:17Z)
• VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval [56.12310817934239]
  Cross-modal embeddings behave as bags of concepts and underrepresent structured visual relationships such as pose and viewpoint.<n>We propose Visualize-then-Retrieve (VisRet), a new paradigm for T2I retrieval that mitigates this limitation of cross-modal similarity alignment.<n>VisRet substantially outperforms cross-modal similarity matching and baselines that recast T2I retrieval as text-to-text similarity matching.
  arXiv  Detail & Related papers  (2025-05-26T17:59:33Z)
• Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation [103.90033029330527]
  Few-Shot Instance (FSIS) requires detecting and segmenting novel classes with limited support examples. We introduce a unified framework, Reference Twice (RefT), to exploit the relationship between support and query features for FSIS.
  arXiv  Detail & Related papers  (2023-01-03T15:33:48Z)
• Decoupled and Memory-Reinforced Networks: Towards Effective Feature Learning for One-Step Person Search [65.51181219410763]
  One-step methods have been developed to handle pedestrian detection and identification sub-tasks using a single network. There are two major challenges in the current one-step approaches. We propose a decoupled and memory-reinforced network (DMRNet) to overcome these problems.
  arXiv  Detail & Related papers  (2021-02-22T06:19:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.