Dynamic Linear Attention

• URL: http://arxiv.org/abs/2606.10650v1
• Date: Tue, 09 Jun 2026 09:57:48 GMT
• Title: Dynamic Linear Attention
• Abstract summary: We propose DLA, a dynamic memory modeling framework for multi-state linear attention.<n>We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories.
• Score: 33.7476613423211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.

Related papers

• Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations [8.050927352582827]
  Transformers lack a native lookup mechanism, requiring repeated dense computation to recognize and reuse local static patterns.<n>Lngram v1 introduces tokenizer-independent conditional memory through discrete latent n-gram addressing.<n>Lngram v2 decouples the number of routes, memory dimension, and backbone width.
  arXiv  Detail & Related papers  (2026-09-03T06:33:23Z)
• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity [34.491686452909526]
  We introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme.<n>We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks.
  arXiv  Detail & Related papers  (2026-07-08T13:17:19Z)
• SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models [75.75091459417304]
  Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding.<n>LVLMs incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices.<n>We propose a binarization method to drastically reduce storage and computational costs.
  arXiv  Detail & Related papers  (2026-07-02T08:30:00Z)
• Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams [78.9560820155271]
  We evaluate online adaptation over streaming, continually updating knowledge.<n> OAKS comprises two datasets: OAKS-BABI and OAKS-Novel.<n>Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS.
  arXiv  Detail & Related papers  (2026-03-08T00:38:39Z)
• MambaX: Image Super-Resolution with State Predictive Control [48.76194230142064]
  Mamba has emerged as a promising approach that can represent the entire reconstruction process as a state sequence with multiple nodes, allowing for intermediate intervention.<n>We created a nonlinear state predictive control model textbfMambaX that maps consecutive spectral bands into a latent state space and generalizes the SR task by dynamically learning the nonlinear state parameters of control equations.<n>Our evaluation demonstrates the superior performance of the dynamic spectrum-state representation model in both single-image SR and multimodal fusion-based SR tasks.
  arXiv  Detail & Related papers  (2025-11-22T11:44:09Z)
• Higher-order Linear Attention [59.92962330635185]
  quadratic cost of scaled dot-product attention is a central obstacle to scaling autoregressive language models to long contexts.<n>We introduce Higher-order Linear Attention (HLA), a causal, streaming mechanism that realizes higher interactions via compact prefix sufficient statistics.
  arXiv  Detail & Related papers  (2025-10-31T07:54:37Z)
• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention [53.22963042513293]
  Large language models (LLMs) excel at capturing global token dependencies via self-attention but face prohibitive compute and memory costs on lengthy inputs.<n>We first propose dual-state linear attention (A), a novel design that maintains two hidden states-one for preserving historical context and one for tracking recencythereby mitigating the short-range bias typical of linear-attention architectures.<n>We introduce DSLA-Serve, an online adaptive distillation framework that progressively replaces Transformer layers DSLA layers at inference time, guided by a sensitivity-based layer ordering.
  arXiv  Detail & Related papers  (2025-06-11T01:25:06Z)
• CD-NGP: A Fast Scalable Continual Representation for Dynamic Scenes [31.783117836434403]
  CD-NGP is a continual learning framework that reduces memory overhead and enhances scalability.<n>It significantly reduces training memory usage to 14GB and requires only 0.4MB/frame in streaming bandwidth on DyNeRF.
  arXiv  Detail & Related papers  (2024-09-08T17:35:48Z)
• Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences [60.489682735061415]
  We propose CHELA, which replaces state space models with short-long convolutions and implements linear attention in a divide-and-conquer manner. Our experiments on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-06-12T12:12:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.