RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention

• URL: http://arxiv.org/abs/2609.07306v1
• Date: Mon, 07 Sep 2026 10:23:06 GMT
• Title: RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention
• Abstract summary: RouteRelay is a router-agnostic method that reuses only route metadata across depth.<n>We show that RouteRelay retains at least 99.99% route recall while rerouting 25.0%, 55.4%, and 78.2% of rows under low, moderate, and high cross-layer drift.
• Score: 10.543370808241159
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dynamic sparse attention reduces long-context prefill cost by routing each query chunk to a small set of key chunks at every Transformer layer. The sparse attention kernel avoids most token interactions, but the router still rebuilds a chunk--chunk score matrix layer after layer, even when the selected routes change little. We introduce RouteRelay, a router-agnostic method that reuses only route metadata across depth while continuing to compute attention with the current layer's queries, keys, and values. Anchor layers perform full routing. Intermediate layers rescore the previous top-$k$ route and a compact sentinel set of near-miss and randomly probed chunks. A query row is rerouted only when a sentinel challenges its weakest selected chunk. We give a top-$k$ stability condition, a probabilistic bound on missed challengers, and a row-selective GPU execution design. In a reproducible empirical evaluation, RouteRelay retains at least 99.99% route recall while rerouting 25.0%, 55.4%, and 78.2% of rows under low, moderate, and high cross-layer drift, respectively. Across routing scales, RouteRelay retains 100.0% recall while evaluating 38.4--51.6% of full-routing score pairs as the key-chunk count grows from 128 to 1024. Its unfused CPU execution remains slower than dense matrix multiplication, exposing row compaction and ledger updates as the main kernel-engineering targets.

Related papers

• LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference [0.0]
  LayerRoute is a parameter-efficient method for adaptive transformer layer-skipping.<n>It combines per-layer hard-gated routing with joint LoRA fine-tuning.<n>It trains in under 7 minutes on a single A100 and adds negligible overhead.
  arXiv  Detail & Related papers  (2026-09-12T03:30:59Z)
• Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference [13.935681015472262]
  Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens.<n>Existing methods treat each routing decision as a local operation conditioned solely on the current hidden state.<n>We propose History-Aware Routing (HeRo), a dynamic routing framework that maintains an explicit routing state across model depth.
  arXiv  Detail & Related papers  (2026-09-08T03:20:07Z)
• Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts [0.0]
  We show that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems.<n>A matched-rank comparison shows that residual representations are often easier to predict across layers, while router-control states preserve the model's expert choices much more faithfully.
  arXiv  Detail & Related papers  (2026-09-02T10:17:12Z)
• CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing [90.63350360476294]
  We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm.<n>We show that the routing decision can be read directly off the structure of the proxy attention map.<n>We formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts.
  arXiv  Detail & Related papers  (2026-09-01T22:49:46Z)
• MACRO: Markov Chain Routing of Transformer Layers [10.52527481709651]
  MACRO is a framework that learns task-specific routes over LLM architectures without modifying underlying parameters.<n>We evaluate MACRO across diverse reasoning and knowledge benchmarks on multiple open-weight LLMs.<n>We outperform the best dynamic routing approach Dr. LLM by +7.2%, while reducing route-search time 9.4x (from 14.8 to 1.6 hours)
  arXiv  Detail & Related papers  (2026-08-06T10:51:34Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• You Only Index Once: Cross-Layer Sparse Attention with Shared Routing [61.29627714699688]
  Cross-layer sparse attention (A) is built on top of KV-sharing architectures such as YOCO.<n>A single indexer computes token-level top-k selection once and reuses the resulting index across layers.<n>Results show CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context.
  arXiv  Detail & Related papers  (2026-06-04T17:54:04Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• End-to-End Sensitivity-Based Filter Pruning [49.61707925611295]
  We present a sensitivity-based filter pruning algorithm (SbF-Pruner) to learn the importance scores of filters of each layer end-to-end. Our method learns the scores from the filter weights, enabling it to account for the correlations between the filters of each layer.
  arXiv  Detail & Related papers  (2022-04-15T10:21:05Z)
• Sequential Routing Framework: Fully Capsule Network-based Speech Recognition [5.730259752695884]
  This paper presents a sequential routing framework to adapt a CapsNet-only structure to sequence-to-sequence recognition. It achieves a 1.1% lower word error rate at 16.9% on the Wall Street Journal corpus. It attains a 0.7% lower phone error rate at 17.5% compared to convolutional neural network-based CTC networks.
  arXiv  Detail & Related papers  (2020-07-23T01:51:41Z)
• Experiments on route choice set generation using a large GPS trajectory set [9.139729923546247]
  In this paper, different path generation algorithms are evaluated using a large GPS trajectory dataset. An initial analysis is performed by generating a single route based on the shortest path. A modified link penalty method, which accounts for preference of using higher hierarchical roads, provides a route set with 97% coverage.
  arXiv  Detail & Related papers  (2020-05-22T13:44:49Z)
• Capsules with Inverted Dot-Product Attention Routing [84.89818784286953]
  We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. Our method improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks.
  arXiv  Detail & Related papers  (2020-02-12T02:09:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.