FLARE++: Low-rank attention with dynamic attention routing

• URL: http://arxiv.org/abs/2608.11519v1
• Date: Wed, 12 Aug 2026 00:16:46 GMT
• Title: FLARE++: Low-rank attention with dynamic attention routing
• Abstract summary: Full self-attention is a strong token mixer for PDE surrogates on irregular domains, but its quadratic cost limits its use on high-resolution problems.<n> Efficient latent-attention models such as the Fast Low-rank Attention Engine (FLARE) avoid that cost by routing all N tokens through M N learned latent queries.<n>We remove this restriction with FLARE++, a low-rank attention architecture with dynamic token routing.
• Score: 1.2425910171551517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Full self-attention is a strong token mixer for PDE surrogates on irregular domains, but its quadratic cost limits its use on high-resolution problems. Efficient latent-attention models such as the Fast Low-rank Attention Routing Engine (FLARE) avoid that cost by routing all N tokens through M << N learned latent queries, but those queries are parameters: once trained, the same learned query templates serve every input. We remove this restriction with FLARE++, a low-rank attention architecture with dynamic token routing. FLARE++ reuses FLARE's own encoder to build its routing queries: learned latent seeds drive one extra encode call that gathers the N input tokens into M input-conditioned queries, and those queries then determine how the same tokens are compressed and redistributed. This preserves FLARE's explicit low-rank factorization and linear O(NM) complexity, and expresses the complete routing operation with standard scaled dot-product attention (SDPA) calls alone. We also provide a multi-GPU context-parallel implementation that shards input tokens across devices without ever gathering the full token sequence on one of them. FLARE++ is competitive across a set of standard PDE surrogate benchmarks, improving on fixed-query FLARE by 24% on average, and it gains 2.3 points of average accuracy on Long Range Arena.

Related papers

• LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference [63.932718076408584]
  LAST is a training-free framework for query-dependent visual token pruning in edge-cloud collaborative MLLM inference.<n>It preserves 95.4% of the full-token accuracy while retaining only 12.5% of the visual tokens, with low edge-side selection overhead and reduced cloud-side computation.
  arXiv  Detail & Related papers  (2026-07-30T09:59:25Z)
• How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs [52.712343092801326]
  FlowTracer is an RL framework that traces answer-targeted reasoning flow on an attention-weighted directed acyclic graph.<n>It scores tokens by flow throughput, revealing high-impact hubs and aggregation checkpoints that mediate long-range dependencies.<n>These derived importances are used to shape token-level rewards, enabling learning signals to focus precisely on the tokens that route information toward correct answers.
  arXiv  Detail & Related papers  (2026-06-09T09:56:51Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning [14.508049757681862]
  LLM-powered coding agents spend the majority of their token budget reading repository files.<n>We show that this formulation creates a modeling bottleneck.<n>We propose LaMR, a structured pruning framework that decomposes code relevance into two interpretable quality dimensions.
  arXiv  Detail & Related papers  (2026-05-14T18:30:10Z)
• TIDE: Every Layer Knows the Token Beneath the Context [21.609102126208608]
  We propose TIDE, which augments the standard transformer with EmbeddingMemory.<n>We theoretically and empirically establish the benefits of TIDE in addressing the issues associated with single-token identity injection.
  arXiv  Detail & Related papers  (2026-05-07T13:16:18Z)
• DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention [2.7422645382944935]
  We present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only salient tokens.<n>Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput.
  arXiv  Detail & Related papers  (2026-03-09T07:02:01Z)
• Residual Context Diffusion Language Models [90.07635240595926]
  Residual Context Diffusion (RCD) is a module that converts discarded token representations into contextual residuals and injects them back for the next denoising step.<n>RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead.
  arXiv  Detail & Related papers  (2026-01-30T13:16:32Z)
• DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning [6.468843780300177]
  We present textbfDELTA, a training-free sparse attention mechanism that achieves computational efficiency without sacrificing model accuracy.<n>Our results show that selective reuse of intermediate attention maps offers a robust path toward efficient long-context reasoning.
  arXiv  Detail & Related papers  (2025-10-10T21:37:49Z)
• SparseD: Sparse Attention for Diffusion Language Models [98.05780626106555]
  diffusion language models (DLMs) offer a promising alternative to autoregressive models (ARs)<n>Existing open-source DLMs suffer from high inference latency.<n>We propose SparseD, a novel sparse attention method for DLMs.
  arXiv  Detail & Related papers  (2025-09-28T18:10:10Z)
• FLARE: Fast Low-rank Attention Routing Engine [2.9498261840961777]
  Fast Low-rank Attention Engine (FLARE) is a linear complexity self-attention mechanism that routes attention through fixed-length latent sequences.<n>FLARE not only scales to unprecedented problem sizes, but also delivers superior accuracy compared to state-of-the-art neural PDE surrogates.
  arXiv  Detail & Related papers  (2025-08-18T03:00:55Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention [43.211427581302715]
  We propose Hierarchically Pruned Attention (HiP) to increase context length in large language models.<n>HiP reduces the time complexity of the attention mechanism to $O(T log T)$ and the space complexity to $O(T)$, where $T$ is the sequence length.<n>We show that HiP significantly reduces both prefill and decoding latencies, as well as memory usage, while maintaining high-quality generation with minimal degradation.
  arXiv  Detail & Related papers  (2024-06-14T08:32:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.