CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones

• URL: http://arxiv.org/abs/2509.21764v1
• Date: Fri, 26 Sep 2025 01:59:29 GMT
• Title: CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones
• Abstract summary: ViT backbones adopt spatial architectural designs, such as window attention, relative positional embeddings in SAM, and RoPE in DINOv3.<n>We introduce a simple yet effective token merging method that maintains spatial integrity, enabling seamless compatibility with spatial architectures.<n>Our approach employs (i)a 2D reduction strategy to enforce structured token layouts, (ii)a spatial-aware merging algorithm that maintains relative token positions, and (iii)a novel max-magnitude-per-dimension token representation.
• Score: 0.7448254811651417
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many modern ViT backbones adopt spatial architectural designs, such as window attention, decomposed relative positional embeddings in SAM, and RoPE in DINOv3. Such architectures impose new challenges on token reduction, as the vast majority of existing methods fail to preserve the spatial structure these architectures depend on. In this paper, we introduce a simple yet effective token merging method that maintains spatial integrity, enabling seamless compatibility with spatial architectures. We reconcile two seemingly conflicting requirements: (i)exploiting the uneven information distribution across the spatial layout while (ii)preserving the spatial structure post-merging. Our approach employs (i)a 2D reduction strategy to enforce structured token layouts, (ii)a spatial-aware merging algorithm that maintains relative token positions, and (iii)a novel max-magnitude-per-dimension token representation that preserves salient features. Our method demonstrates strong performance both off-the-shelf and with fine-tuning, achieving state-of-the-art results on spatial and non-spatial architectures across various vision tasks. Specifically, we achieve 1.25x speedup on SAM-H with only 0.7% mIOU drop evaluated on COCO off-the-shelf, and 1.15x speedup on DeiT-B with no top-1 accuracy drop on ImageNet within just one epoch of fine-tuning.

Related papers

• Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency [71.81785123423516]
  We propose a task-specific pruning pipeline, named Cut-ViT, for pruning visual foundation models.<n> Basis-agnostic and residual constraints are adopted to align the gram subspaces between the native and pruned DINOv3 models.<n>Experiments show that Cut-ViT requires approximately one minute on a single A100 GPU to obtainworks at various sparsity levels.
  arXiv  Detail & Related papers  (2026-08-28T11:25:36Z)
• SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video [88.79625979053873]
  We propose a novel framework, SpaceEra, for pre-trained vision-language models (VLMs)<n>We extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference.<n>In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations.
  arXiv  Detail & Related papers  (2026-07-02T06:56:29Z)
• Spatial-Aware Reduction Framework: Towards Efficient and Faithful Visual State Space Models [71.59431204970339]
  STORM is a spatial-aware token reduction framework.<n>It reformulates reduction into a structured operation on spatial units.<n>It achieves state-of-the-art pruning accuracy across diverse vision Mamba backbones.
  arXiv  Detail & Related papers  (2026-06-18T08:31:11Z)
• LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory [97.14005794889134]
  We present LoGeR, a novel architecture that scales dense 3D reconstruction to extremely long sequences without post-optimization.<n>LoGeR processes video streams in chunks, leveraging strong bidirectional priors for high-fidelity intra-chunk reasoning.<n>This memory architecture enables LoGeR to be trained on sequences of 128 frames, and generalize up to thousands of frames during inference.
  arXiv  Detail & Related papers  (2026-03-03T18:55:37Z)
• OCTOPUS: Enhancing the Spatial-Awareness of Vision SSMs with Multi-Dimensional Scans and Traversal Selection [20.717476762904038]
  We introduce OCTOPUS, a novel architecture that preserves both global context and local spatial structure within images.<n>OCTOPUS performs discrete reoccurrence along eight principal orientations, going forward or backward in the horizontal, vertical, and diagonal directions.<n>In our classification and segmentation benchmarks, OCTOPUS demonstrates notable improvements in boundary preservation and region consistency.
  arXiv  Detail & Related papers  (2026-01-31T21:12:59Z)
• CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval [54.15776146365823]
  Composed Image Retrieval (CIR) enables users to search for target images using both a reference image and manipulation text.<n>We propose CSMCIR, a unified representation framework that achieves efficient query-target alignment through three synergistic components.
  arXiv  Detail & Related papers  (2026-01-07T09:21:38Z)
• SVRecon: Sparse Voxel Rasterization for Surface Reconstruction [60.92372415355283]
  We extend the recently proposed sparse voxelization paradigm to the task of high-fidelity surface reconstruction by integrating SVRecon.<n>Our method achieves strong reconstruction accuracy while having consistently speedy convergence.
  arXiv  Detail & Related papers  (2025-11-21T16:32:01Z)
• CAGE: Continuity-Aware edGE Network Unlocks Robust Floorplan Reconstruction [6.027087091285519]
  textbfCAGE (textitContinuity-Aware edGE) network reconstructs vector floorplans directly from point-cloud density maps.<n>textbfCAGE achieves state-of-the-art performance, with F1 scores of 99.1% (rooms), 91.7% (corners), and 89.3% (angles)
  arXiv  Detail & Related papers  (2025-09-18T22:10:37Z)
• Learning Spatial Decay for Vision Transformers [50.63391799053993]
  Vision Transformers (ViTs) have revolutionized computer vision, yet their self-attention mechanism lacks explicit spatial inductive biases.<n>Existing approaches introduce data-independent spatial decay based on fixed distance metrics.<n>We present the first successful adaptation of data-dependent spatial decay to 2D vision transformers.
  arXiv  Detail & Related papers  (2025-08-13T06:18:32Z)
• Identity-Preserving Text-to-Video Generation Guided by Simple yet Effective Spatial-Temporal Decoupled Representations [131.33758144860988]
  Identity-preserving text-to-video (IPT2V) generation aims to create high-fidelity videos with consistent human identity.<n>Current end-to-end frameworks suffer a critical spatial-temporal trade-off.<n>We propose a simple yet effective spatial-temporal decoupled framework that decomposes representations into spatial features for layouts and temporal features for motion dynamics.
  arXiv  Detail & Related papers  (2025-07-07T06:54:44Z)
• EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality [16.576495786546612]
  We introduce Efficient Vision Mamba, a novel architecture built on hidden state mixer-based state space duality (HSM-SSD)<n>We propose multi-stage hidden state fusion to reinforce the representation power of hidden states and provide the design to alleviate the bottleneck caused by the memory-bound operations.<n>As a result, the EfficientViM family achieves a new state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7% performance improvement over the second-best model SHViT with faster speed.
  arXiv  Detail & Related papers  (2024-11-22T02:02:06Z)
• Searching a High-Performance Feature Extractor for Text Recognition Network [92.12492627169108]
  We design a domain-specific search space by exploring principles for having good feature extractors. As the space is huge and complexly structured, no existing NAS algorithms can be applied. We propose a two-stage algorithm to effectively search in the space.
  arXiv  Detail & Related papers  (2022-09-27T03:49:04Z)
• Joint Spatial-Temporal and Appearance Modeling with Transformer for Multiple Object Tracking [59.79252390626194]
  We propose a novel solution named TransSTAM, which leverages Transformer to model both the appearance features of each object and the spatial-temporal relationships among objects. The proposed method is evaluated on multiple public benchmarks including MOT16, MOT17, and MOT20, and it achieves a clear performance improvement in both IDF1 and HOTA.
  arXiv  Detail & Related papers  (2022-05-31T01:19:18Z)
• Index $t$-SNE: Tracking Dynamics of High-Dimensional Datasets with Coherent Embeddings [1.7188280334580195]
  This paper presents a methodology to reuse an embedding to create a new one, where cluster positions are preserved. The proposed algorithm has the same complexity as the original $t$-SNE to embed new items, and a lower one when considering the embedding of a dataset sliced into sub-pieces.
  arXiv  Detail & Related papers  (2021-09-22T06:45:37Z)
• CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification [102.89434996930387]
  VI-ReID aims to match cross-modality pedestrian images, breaking through the limitation of single-modality person ReID in dark environment. Existing works manually design various two-stream architectures to separately learn modality-specific and modality-sharable representations. We propose a novel method, named Cross-Modality Neural Architecture Search (CM-NAS)
  arXiv  Detail & Related papers  (2021-01-21T07:07:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.