Dual Dimensionality for Local and Global Attention

• URL: http://arxiv.org/abs/2606.18587v1
• Date: Wed, 17 Jun 2026 01:27:33 GMT
• Title: Dual Dimensionality for Local and Global Attention
• Abstract summary: We show that local and distant tokens impose asymmetric demands on representational capacity.<n>We formalize this idea as Distance-Adaptive Representation (DAR), implemented in a controlled setting.<n>DAR preserves full-dimensional representations within a local context window while assigning reduced-dimensional representations to tokens beyond that window.
• Score: 25.528738579381994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its distance from the prediction target. In natural language, however, the next word is most strongly influenced by the immediately preceding tokens. We hypothesize that local and distant tokens impose asymmetric demands on representational capacity: local tokens are more critical for predicting immediate outputs and thus require richer representations, whereas distant tokens primarily serve as long-range memory, for which lower-dimensional representations may suffice. We formalize this idea as Distance-Adaptive Representation (DAR), implemented in a controlled setting that preserves full-dimensional representations within a local context window while assigning reduced-dimensional representations (e.g. 1/4 of the original dimensionality) to tokens beyond that window. Across multiple pretraining scales (70M to 410M parameters), as well as continued supervised fine-tuning on a 1B-scale model, this approach closely matches the performance of full-dimensional baselines. In contrast, uniformly reducing dimensionality across all token positions leads to worse performance. These results challenge the common assumption that key and value dimensionality should be uniform across token positions. Our findings suggest a new direction for designing attention architectures that adaptively allocate representational capacity across sequences, enabling further reductions in KV cache during inference.

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• PointSplat: Compact Gaussian Splatting via Human-Centric Prediction [70.45838533838104]
  We propose PointSplat, a novel human-centric approach to producing 3D human representations from input views on the fly.<n>The proposed method first estimates a coarse geometric proxy and performs ray casting to prune redundant points.<n>It then employs a Point-Image Transformer to fuse appearance and geometry features, predicting Gaussian attributes in a single forward pass.
  arXiv  Detail & Related papers  (2026-06-30T17:59:05Z)
• Representative Attention For Vision Transformers [11.0298065364767]
  Linear attention has emerged as a promising direction for scaling Vision Transformers beyond the quadratic cost of dense self-attention.<n>We propose Representative Attention (RPAttention), a linear global attention mechanism that performs token compression directly in representation space.<n>RPAttention reduces the dominant token interaction complexity from quadratic to linear scaling with respect to the number of spatial tokens.
  arXiv  Detail & Related papers  (2026-05-14T14:48:20Z)
• Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens [88.42820935044021]
  We presentCub Discrete Diffusion (CubiD), the first discrete generation model for high-dimensional representations.<n>CubiD performs fine-grained masking throughout the high-dimensional discrete representation.<n>On ImageNet-256, CubiD achieves state-of-the-art discrete generation with strong scaling behavior from 900M to 3.7B parameters.
  arXiv  Detail & Related papers  (2026-03-19T17:59:55Z)
• ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation [64.84095852784714]
  Residual Tokenizer (ResTok) is a 1D visual tokenizer that builds hierarchical residuals for both image tokens and latent tokens.<n>We show that restoring hierarchical residual priors in visual tokenization significantly improves AR image generation, achieving a gFID of 2.34 on ImageNet-256 with only 9 sampling steps.
  arXiv  Detail & Related papers  (2026-01-07T14:09:18Z)
• Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space [56.37266873329401]
  Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density.<n>We propose $textbfDynamic Large Concept Models (DLCM)$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts from tokens to a compressed concept space where reasoning is more efficient.
  arXiv  Detail & Related papers  (2025-12-31T04:19:33Z)
• Neighboring Autoregressive Modeling for Efficient Visual Generation [19.486745219466666]
  Neighboring Autoregressive Modeling (NAR) is a novel paradigm that formulates autoregressive visual generation as a progressive outpainting procedure.<n>To enable parallel prediction of multiple adjacent tokens in the spatial-temporal space, we introduce a set of dimension-oriented decoding heads.<n>Experiments on ImageNet$256times 256$ and UCF101 demonstrate that NAR achieves 2.4$times$ and 8.6$times$ higher throughput respectively.
  arXiv  Detail & Related papers  (2025-03-12T05:52:27Z)
• Quantize What Counts: More for Keys, Less for Values [63.51476878610841]
  Large Language Models (LLMs) suffer inference-time memory bottlenecks dominated by the attention Key-Value ( KV) cache.<n>This paper proposes two theorems that anchor mixed-precision KV quantization in the intrinsic geometry of Transformer models.
  arXiv  Detail & Related papers  (2025-02-20T22:24:27Z)
• Attamba: Attending To Multi-Token States [6.5676809841642125]
  We introduce Attamba, a novel architecture that uses state-space models to compress chunks of tokens. We find that replacing key and value projections in a transformer with SSMs can improve model quality and enable flexible token chunking. Attamba can perform attention on chunked-sequences of variable length, enabling a smooth transition between quadratic and linear scaling.
  arXiv  Detail & Related papers  (2024-11-26T18:52:06Z)
• Transformers are Universal In-context Learners [21.513210412394965]
  We show that deep transformers can approximate continuous in-context mappings to arbitrary precision, uniformly over compact token domains. A key aspect of our results, compared to existing findings, is that for a fixed precision, a single transformer can operate on an arbitrary (even infinite) number of tokens.
  arXiv  Detail & Related papers  (2024-08-02T16:21:48Z)
• AiluRus: A Scalable ViT Framework for Dense Prediction [95.1313839257891]
  Vision transformers (ViTs) have emerged as a prevalent architecture for vision tasks owing to their impressive performance. We propose to apply adaptive resolution for different regions in the image according to their importance. We evaluate our proposed method on three different datasets and observe promising performance.
  arXiv  Detail & Related papers  (2023-11-02T12:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.