Mixture-of-Depths Attention

• URL: http://arxiv.org/abs/2603.15619v1
• Date: Mon, 16 Mar 2026 17:59:55 GMT
• Title: Mixture-of-Depths Attention
• Abstract summary: Scaling depth is a key driver for large language models (LLMs)<n>We introduce mixture-of-depths attention (MoDA)<n>MoDA allows each attention head to attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers.
• Score: 65.80640499676542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling depth is a key driver for large language models (LLMs). Yet, as LLMs become deeper, they often suffer from signal degradation: informative features formed in shallow layers are gradually diluted by repeated residual updates, making them harder to recover in deeper layers. We introduce mixture-of-depths attention (MoDA), a mechanism that allows each attention head to attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers. We further describe a hardware-efficient algorithm for MoDA that resolves non-contiguous memory-access patterns, achieving 97.3% of FlashAttention-2's efficiency at a sequence length of 64K. Experiments on 1.5B-parameter models demonstrate that MoDA consistently outperforms strong baselines. Notably, it improves average perplexity by 0.2 across 10 validation benchmarks and increases average performance by 2.11% on 10 downstream tasks, with a negligible 3.7% FLOPs computational overhead. We also find that combining MoDA with post-norm yields better performance than using it with pre-norm. These results suggest that MoDA is a promising primitive for depth scaling. Code is released at https://github.com/hustvl/MoDA .

Related papers

• Output Dilution: Redundant but Fragile Representations in MoE Models [1.0829694003408499]
  Mixture-of-Experts (MoE) models appear to encode moral content as robustly as dense models, yet prove far more fragile in their encoding.<n>In OLMoE-1B-7B, linear probes recover moral valence from nearly every expert-layer combination, with mean peak-layer accuracy above 90%.<n>But these representations collapse under levels of activation noise that a dense model of matched size easily tolerates, with a 4.2-fold difference in robustness.<n>Because the MoE block averages across active experts before contributing to the residual stream, the feedforward signal reaching downstream layers is nearly two orders of magnitude smaller than in
  arXiv  Detail & Related papers  (2026-08-25T23:41:07Z)
• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models [129.64558694024126]
  Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling.<n>We characterize how optimization hyper parameters, compute allocation, and architecture scale for MoE dLLMs.<n>We train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens.
  arXiv  Detail & Related papers  (2026-08-04T10:53:02Z)
• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks [0.09665867043802821]
  Auto-regressive Large Language Models (LLMs) achieve strong performance on coding tasks, but incur high memory and inference costs.<n>We investigate the application and robustness of PTQ techniques, specifically GPTQ and a modified Hessian-Aware Quantization (HAWQ) algorithm.
  arXiv  Detail & Related papers  (2026-04-22T00:53:43Z)
• LLaDA2.1: Speeding Up Text Diffusion via Token Editing [72.92893150459909]
  We unveil LLaDA2.1, a paradigm shift designed to transcend the trade-off between decoding speed and generation quality.<n>By seamlessly weaving Token-to-Token (T2T) editing into the conventional Mask-to-Token (M2T) scheme, we introduce a joint, threshold-decoding scheme.<n>This structural innovation gives rise to two distinct personas: the Speedy Mode (S Mode), which audaciously lowers the M2T threshold to bypass traditional constraints while relying on T2T to refine the output; and the Quality Mode (Q Mode), which leans into conservative thresholds to secure superior benchmark
  arXiv  Detail & Related papers  (2026-02-09T14:00:07Z)
• Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference [9.469995152350899]
  We propose Kascade, a training-free sparse attention method that leverages known observations.<n>Kascade computes exact Top-k indices in a small set of anchor layers, then reuses those indices in intermediate reuse layers.<n>Kascade achieves up to 4.1x speedup in decode attention and 2.2x speedup in prefill attention over FlashAttention-3 baseline on H100 GPUs.
  arXiv  Detail & Related papers  (2025-12-18T10:37:14Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [52.079202872069835]
  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs) have grown rapidly in size.<n>We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression [22.038650467915176]
  We propose the Mixture of Attention (MoA), which automatically tailors distinct sparse attention configurations to different heads and layers. MoA adapts to varying input sizes, revealing that some attention heads expand their focus to accommodate longer sequences, while other heads consistently concentrate on fixed-length local contexts.
  arXiv  Detail & Related papers  (2024-06-21T06:58:37Z)
• Improving Dual-Encoder Training through Dynamic Indexes for Negative Mining [61.09807522366773]
  We introduce an algorithm that approximates the softmax with provable bounds and that dynamically maintains the tree. In our study on datasets with over twenty million targets, our approach cuts error by half in relation to oracle brute-force negative mining.
  arXiv  Detail & Related papers  (2023-03-27T15:18:32Z)
• An efficient encoder-decoder architecture with top-down attention for speech separation [25.092542427133704]
  We provide a bio-inspired efficient encoder-decoder architecture by mimicking the brain's top-down attention, called TDANet. On three benchmark datasets, TDANet consistently achieved competitive separation performance to previous state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2022-09-30T03:09:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.