Sequential Token Merging: Revisiting Hidden States

• URL: http://arxiv.org/abs/2509.22691v1
• Date: Fri, 19 Sep 2025 14:45:01 GMT
• Title: Sequential Token Merging: Revisiting Hidden States
• Abstract summary: Vision Mambas (ViMs) achieve remarkable success with sub-quadratic complexity, but their efficiency remains constrained by quadratic token scaling with image resolution.<n>We propose Sequential Token Merging (STM), featuring: 1) Bidirectional nearest neighbor merging to preserve sequential dependencies through symmetric spatial aggregation, and 2) Hidden states protection to stabilize the hidden states around the class token.<n>Our method achieves state-of-the-art efficiency with minimal complexity, while providing new insights into state-space model dynamics.
• Score: 29.89737086314315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Mambas (ViMs) achieve remarkable success with sub-quadratic complexity, but their efficiency remains constrained by quadratic token scaling with image resolution. While existing methods address token redundancy, they overlook ViMs' intrinsic Limited Directional Sequential Dependence (LDSD) - a critical information flow mechanism revealed in our analysis. We further identify Mamba's selective scan enables gradual information aggregation in hidden states. Based on these insights, we propose Sequential Token Merging (STM), featuring: 1) Bidirectional nearest neighbor merging to preserve sequential dependencies through symmetric spatial aggregation, and 2) Hidden states protection to stabilize the hidden states around the class token. STM strategically leverages Mamba's layer-wise loss convergence to convert temporal forgetfulness into stability. Experiments demonstrate STM's superiority: 1.0% accuracy drop for ViM-Ti at 20% token reduction, and only 1.4% degradation for ViM-S at 40% reduction. Our method achieves state-of-the-art efficiency with minimal complexity, while providing new insights into state-space model dynamics. Codes will be released soon.

Related papers

• AdaSurvMamba: Dynamic Fusion and Semantic Scanning for Multimodal Survival Analysis [57.942670844512016]
  We introduce AdaSurvMamba as a novel adaptive framework for multimodal survival analysis.<n>The framework features a Dual-Scale Importance-Aware Reconstruction (DSIR) module to dynamically modulate cross-modal interaction strength.<n>Experiments across five TCGA cohorts demonstrate consistent gains over existing methods.
  arXiv  Detail & Related papers  (2026-06-28T11:36:31Z)
• Spatial-Aware Reduction Framework: Towards Efficient and Faithful Visual State Space Models [71.59431204970339]
  STORM is a spatial-aware token reduction framework.<n>It reformulates reduction into a structured operation on spatial units.<n>It achieves state-of-the-art pruning accuracy across diverse vision Mamba backbones.
  arXiv  Detail & Related papers  (2026-06-18T08:31:11Z)
• SparseSAM: Structured Sparsification of Activations in Segment Anything Models [26.589924688727795]
  Segment Anything Model (SAM) achieves strong open-vocabulary segmentation, but its ViT-based image encoders dominate latency inference and memory.<n>We propose SparseSAM, a training-free sparsification framework that jointly accelerates attention and layers while preserving token identity.<n>Across four segmentation benchmarks, SparseSAM loses only 0.004 mIoU at a density and 0.021 mIoU at 0.3, a 2.10x reduction in accuracy loss versus token merging advances, while achieving 2x faster inference and 2.8x memory reduction.
  arXiv  Detail & Related papers  (2026-05-17T19:54:22Z)
• StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models [57.674757786328236]
  StructSAM is a resolution-preserving merge-unmerge framework tailored to Segment Anything Model (SAM)<n>We show that StructSAM reduces encoder FLOPs by 25-30% (up to 40%+ with prompt-aware merging) with minor drops in mIoU/Dice.<n>We also provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines.
  arXiv  Detail & Related papers  (2026-03-07T18:30:58Z)
• Stateful Token Reduction for Long-Video Hybrid VLMs [69.6930118088911]
  We study query-conditioned token reduction for hybrid video vision-language models (VLMs)<n>We propose a low-to-high progressive reduction schedule and a unified language-aware scoring mechanism for both attention and Mamba blocks.<n>Under an aggressive compression setting, our approach delivers substantial prefilling speedups with near-baseline accuracy at test time.
  arXiv  Detail & Related papers  (2026-02-27T08:11:06Z)
• Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention [105.11288339285154]
  Efficient-LVSM is a dual-stream architecture that applies intra-view self-attention for input views and self-then-cross attention for target views.<n>It achieves 29.86 dB PSNR on RealEstate10K with 2 input views, surpassing LVSM by 0.2 dB, with 2x faster training convergence and 4.4x faster inference speed.
  arXiv  Detail & Related papers  (2026-02-06T08:11:58Z)
• StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs [54.229363096087866]
  Speech tokenizers are not robust to meaning-irrelevant acoustic perturbations.<n>This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal.<n>We introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism.
  arXiv  Detail & Related papers  (2025-09-26T11:32:51Z)
• Content-Aware Mamba for Learned Image Compression [33.05776457003562]
  We introduce ContentAware Mamba (CAM), an SSM that dynamically adapts its processing to the image content.<n>First, it replaces the rigid scan with a content-adaptive token permutation strategy.<n>Second, it overcomes the sequential dependency by injecting sample-specific global priors into the state-space model.
  arXiv  Detail & Related papers  (2025-08-04T08:42:23Z)
• QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models [8.543701468361395]
  QuarterMap removes redundant spatial activations before scanning and restores dimensions via nearest-neighbor upsampling.<n>On ImageNet-1K, QuarterMap up to 11% speedup on VMamba with less than 0.9% accuracy drop, and yields similar gains on ADE20K segmentation.<n>We validate QuarterMap on MedMamba, a domain-specific model that shares the same four-directional scanning structure.
  arXiv  Detail & Related papers  (2025-07-13T06:49:32Z)
• StruMamba3D: Exploring Structural Mamba for Self-supervised Point Cloud Representation Learning [31.585380521480868]
  We propose StruMamba3D, a novel paradigm for self-supervised point cloud representation learning.<n>We design spatial states and use them as proxies to preserve spatial dependencies among points.<n>Our method attains the SOTA 95.1% accuracy on ModelNet40 and 92.75% accuracy on the most challenging split of ScanObjectNN without voting strategy.
  arXiv  Detail & Related papers  (2025-06-26T17:58:05Z)
• MambaIRv2: Attentive State Space Restoration [96.4452232356586]
  Mamba-based image restoration backbones have recently demonstrated significant potential in balancing global reception and computational efficiency.<n>We propose MambaIRv2, which equips Mamba with the non-causal modeling ability similar to ViTs to reach the attentive state space restoration model.
  arXiv  Detail & Related papers  (2024-11-22T12:45:12Z)
• EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality [16.576495786546612]
  We introduce Efficient Vision Mamba, a novel architecture built on hidden state mixer-based state space duality (HSM-SSD)<n>We propose multi-stage hidden state fusion to reinforce the representation power of hidden states and provide the design to alleviate the bottleneck caused by the memory-bound operations.<n>As a result, the EfficientViM family achieves a new state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7% performance improvement over the second-best model SHViT with faster speed.
  arXiv  Detail & Related papers  (2024-11-22T02:02:06Z)
• MambaVC: Learned Visual Compression with Selective State Spaces [74.29217829932895]
  We introduce MambaVC, a simple, strong and efficient compression network based on SSM. MambaVC develops a visual state space (VSS) block with a 2D selective scanning (2DSS) module as the nonlinear activation function after each downsampling. On compression benchmark datasets, MambaVC achieves superior rate-distortion performance with lower computational and memory overheads.
  arXiv  Detail & Related papers  (2024-05-24T10:24:30Z)
• Graph-Aware Contrasting for Multivariate Time-Series Classification [50.84488941336865]
  Existing contrastive learning methods mainly focus on achieving temporal consistency with temporal augmentation and contrasting techniques. We propose Graph-Aware Contrasting for spatial consistency across MTS data. Our proposed method achieves state-of-the-art performance on various MTS classification tasks.
  arXiv  Detail & Related papers  (2023-09-11T02:35:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.