Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers

• URL: http://arxiv.org/abs/2606.14757v1
• Date: Mon, 08 Jun 2026 09:24:52 GMT
• Title: Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers
• Abstract summary: VIOLIN is an attention mechanism that encodes spatial structure within attention via Space Filling Curves (SFCs)<n>It consistently improves performance across a wide range of evaluations.<n>It can be combined with parameter-efficient fine-tuning methods such as LoRA to further increase the performance.
• Score: 43.297561003640176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Though Vision Transformers (ViTs) have become the dominant backbone in many computer vision tasks, due to permutation equivariance, their attention mechanism lacks explicit spatial inductive biases. This become particularly important in two settings: when model capacity is small or training data is limited. Inspired by the attention masking strategies in Linear Transformers and the scanning patterns of Vision SSMs, we introduce VIOLIN, a lightweight masked attention mechanism that encodes spatial structure within attention via Space Filling Curves (SFCs) with less than 0.0015% extra parameters and negligible computational overhead. VIOLIN scans the image using multiple SFCs to construct curve-specific decay masks, which are then combined and multiplied with the attention matrix. Across a wide range of evaluations, VIOLIN consistently improves performance. In limited data regimes such as fine-tuning on VTAB-1K, it boosts accuracy across all task groups and by up to 8.7% on the tasks where spatial information is essential. It can be combined with parameter-efficient fine-tuning methods such as LoRA to further increase the performance. Beyond fine-tuning, VIOLIN improves various small scale ViT architectures (e.g., DeiT, DINO) during pretraining on ImageNet-1K. Additionally, on pixel-level CIFAR-100 training, a task that is highly dependent on location information, VIOLIN increases accuracy by up to 7.2%. Overall, VIOLIN provides a computationally efficient yet effective way to inject spatial inductive bias into ViTs, especially benefiting small models and limited data settings.

Related papers

• Advancing Vision Transformer with Enhanced Spatial Priors [45.601974887796864]
  Vision Transformer (ViT) has garnered significant attention within the computer vision community.<n>We have proposed RMT, a robust vision backbone with explicit spatial priors for general purposes.<n>Building on the strengths of RMT, Euclidean enhanced Vision Transformer (EVT) is an expanded version that incorporates several key improvements.
  arXiv  Detail & Related papers  (2026-04-20T17:41:00Z)
• Harnessing Lightweight Transformer with Contextual Synergic Enhancement for Efficient 3D Medical Image Segmentation [59.711744386226194]
  We propose Light-UNETR, a lightweight transformer designed to achieve model efficiency.<n>Light-UNETR features a Lightweight Dimension Reductive Attention (LIDR) module, which reduces spatial and channel dimensions.<n>We also introduce a Contextual Synergic Enhancement (CSE) learning strategy, which aims to boost the data efficiency of Transformers.
  arXiv  Detail & Related papers  (2026-03-24T16:24:19Z)
• DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models [45.12546316524245]
  We introduce DART, a fully differentiable Dynamic Region Adaptive Tokenizer.<n>DART employs learnable region scores and quantile-based partitioning to create content-aware patches of varying sizes.<n>The impact of this approach is profound, where a DART-Small matches the performance of a DeiT-Base86 with nearly double the inference speed.
  arXiv  Detail & Related papers  (2025-06-12T06:25:37Z)
• The Linear Attention Resurrection in Vision Transformer [0.6798775532273751]
  Vision Transformers (ViTs) have recently taken computer vision by storm.<n>Softmax attention underlying ViTs comes with a quadratic complexity in time and memory, hindering the application of ViTs to high-resolution images.<n>We propose a linear attention method to address the limitation, which doesn't sacrifice ViT's core advantage of capturing global representation.
  arXiv  Detail & Related papers  (2025-01-27T16:29:17Z)
• Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Heads [10.169639612525643]
  We propose a new multi-head self-attention (MHSA) variant named Fibottention, which can replace MHSA in Transformer architectures.<n>Fibottention is data-efficient and computationally more suitable for processing large numbers of tokens than the standard MHSA.<n>It employs structured sparse attention based on dilated Fibonacci sequences, which, uniquely, differ across attention heads, resulting in-like diverse features across heads.
  arXiv  Detail & Related papers  (2024-06-27T17:59:40Z)
• Vision Transformer with Sparse Scan Prior [24.78780746169092]
  We propose a textbfSparse textbfScan textbfSelf-textbfAttention mechanism ($rmS3rmA$)<n>This mechanism predefines a series of Anchors of Interest for each token and employs local attention to efficiently model the spatial information around these anchors.<n>Building on $rmS3rmA$, we introduce the textbfSparse textbfScan textbfVision
  arXiv  Detail & Related papers  (2024-05-22T04:34:36Z)
• RMT: Retentive Networks Meet Vision Transformers [55.76528783956601]
  Vision Transformer (ViT) has gained increasing attention in the computer vision community in recent years.<n>Self-Attention lacks explicit spatial priors and bears a quadratic computational complexity.<n>We propose RMT, a strong vision backbone with explicit spatial prior for general purposes.
  arXiv  Detail & Related papers  (2023-09-20T00:57:48Z)
• Vicinity Vision Transformer [53.43198716947792]
  We present a Vicinity Attention that introduces a locality bias to vision transformers with linear complexity. Our approach achieves state-of-the-art image classification accuracy with 50% fewer parameters than previous methods.
  arXiv  Detail & Related papers  (2022-06-21T17:33:53Z)
• Pruning Self-attentions into Convolutional Layers in Single Path [89.55361659622305]
  Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. We propose Single-Path Vision Transformer pruning (SPViT) to efficiently and automatically compress the pre-trained ViTs. Our SPViT can trim 52.0% FLOPs for DeiT-B and get an impressive 0.6% top-1 accuracy gain simultaneously.
  arXiv  Detail & Related papers  (2021-11-23T11:35:54Z)
• When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations [111.44860506703307]
  Vision Transformers (ViTs) and existing VisionNets signal efforts on replacing hand-wired features or inductive throughputs with general-purpose neural architectures. This paper investigates ViTs and Res-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and inference. We show that the improved robustness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform Nets of similar size and smoothness when trained from scratch on ImageNet without large-scale pretraining or strong data augmentations.
  arXiv  Detail & Related papers  (2021-06-03T02:08:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.