StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models

• URL: http://arxiv.org/abs/2603.07307v1
• Date: Sat, 07 Mar 2026 18:30:58 GMT
• Title: StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
• Abstract summary: StructSAM is a resolution-preserving merge-unmerge framework tailored to Segment Anything Model (SAM)<n>We show that StructSAM reduces encoder FLOPs by 25-30% (up to 40%+ with prompt-aware merging) with minor drops in mIoU/Dice.<n>We also provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines.
• Score: 57.674757786328236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent token merging techniques for Vision Transformers (ViTs) provide substantial speedups by reducing the number of tokens processed by self-attention, often without retraining. However, their direct application to the Segment Anything Model (SAM) family is nontrivial: SAM's image encoder mixes windowed and global attention, and its mask decoder relies on dense, prompt-conditioned features for precise boundary prediction. We systematically evaluate representative token-merging methods on SAM and Medical SAM in a strict off-the-shelf setting, and find that existing destination-selection heuristics can erode boundaries and leak prompt information as merge rates increase. We propose \textbf{StructSAM}, a resolution-preserving merge-unmerge framework tailored to SAM. StructSAM computes a lightweight token-energy score from first-order feature gradients, uses grid-based flatness screening to protect boundary and prompt regions, and merges tokens within flat areas toward low-energy destinations with explicit token recovery. We further provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines. Across eight natural and medical benchmarks, StructSAM reduces encoder FLOPs by 25-30\% (up to 40\%+ with prompt-aware merging) with minor drops in mIoU/Dice, consistently outperforming ToMe, PiToMe, ToMeSD, VidToMe, and ALGM at the same compute.

Related papers

• EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass [79.69317943751223]
  We propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM)<n>EOVSAM adapts SAM 3 for single-pass prediction.<n>It consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets.
  arXiv  Detail & Related papers  (2026-08-03T14:15:37Z)
• Exploring SAM Supervision for Fine-Grained UAV Target Segmentation under Data Scarcity [0.0]
  Unmanned aerial vehicle (UAV) target segmentation is challenging due to the small size of objects, appearance variations, cluttered backgrounds, and the scarcity of densely annotated data.<n>This paper investigates the use of SAM3 as a pseudo-label generator for training compact segmentation networks.
  arXiv  Detail & Related papers  (2026-07-04T07:59:10Z)
• HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation [61.044699537570644]
  We propose a novel learning framework, named Hierarchical Frequency Prompted SAM (HFP-SAM) for high-performance Marine Animal segmentation.<n>First, we design a Frequency Guided Adapter (FGA) to efficiently inject marine scene information into the frozen SAM backbone.<n>We also introduce a Frequency-aware Point Selection (FPS) to generate highlighted regions through frequency analysis.
  arXiv  Detail & Related papers  (2026-03-13T06:43:26Z)
• A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification [2.0069888187253615]
  Production LLM systems often rely on separate models for safety and other classification-heavy steps.<n>We instead reuse computation already paid for by the serving LLM: we train lightweight probes on its hidden states and predict labels in the same forward pass used for generation.
  arXiv  Detail & Related papers  (2026-01-19T18:40:29Z)
• Granular Computing-driven SAM: From Coarse-to-Fine Guidance for Prompt-Free Segmentation [17.190865623538212]
  We introduce Granular Computing-driven SAM (Grc-SAM), a coarse-to-fine framework motivated by Granular Computing.<n>First, the coarse stage adaptively extracts high-response regions from features to achieve precise foreground localization.<n>Second, the fine stage applies finer patch partitioning with sparse local swin-style attention to enhance detail modeling.<n>Third, refined masks are encoded as latent prompt embeddings for the SAM decoder, replacing handcrafted prompts with an automated reasoning process.
  arXiv  Detail & Related papers  (2025-11-24T12:55:02Z)
• VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel [68.24765319399286]
  We present VesSAM, a powerful and efficient framework tailored for 2D vessel segmentation.<n>VesSAM integrates (1) a convolutional adapter to enhance local texture features, (2) a multi-prompt encoder that fuses anatomical prompts, and (3) a lightweight mask decoder to reduce jagged artifacts.<n>VesSAM consistently outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice and 13% IoU.
  arXiv  Detail & Related papers  (2025-11-02T15:47:05Z)
• CMaP-SAM: Contraction Mapping Prior for SAM-driven Few-shot Segmentation [37.596987175531275]
  Few-shot segmentation (FSS) aims to segment new classes using few annotated images.<n>Recent FSS methods have shown considerable improvements by leveraging Segment Anything Model (SAM)<n>We propose CMaP-SAM, a novel framework that introduces contraction mapping theory to optimize position priors.
  arXiv  Detail & Related papers  (2025-04-07T13:19:16Z)
• SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement [40.37217744643069]
  We propose a universal and efficient approach by adapting SAM to the mask refinement task.<n>Specifically, we introduce a multi-prompt excavation strategy to mine diverse input prompts for SAM.<n>We extend our method to SAMRefiner++ by introducing an additional IoU adaption step to further boost the performance of the generic SAMRefiner on the target dataset.
  arXiv  Detail & Related papers  (2025-02-10T18:33:15Z)
• Bridge the Points: Graph-based Few-shot Segment Anything Semantically [79.1519244940518]
  Recent advancements in pre-training techniques have enhanced the capabilities of vision foundation models. Recent studies extend the SAM to Few-shot Semantic segmentation (FSS) We propose a simple yet effective approach based on graph analysis.
  arXiv  Detail & Related papers  (2024-10-09T15:02:28Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)
• DeSAM: Decoupled Segment Anything Model for Generalizable Medical Image Segmentation [22.974876391669685]
  Segment Anything Model (SAM) shows potential for improving the cross-domain robustness of medical image segmentation. SAM performs significantly worse in automatic segmentation scenarios than when manually prompted. Decoupled SAM modifies SAM's mask decoder by introducing two new modules.
  arXiv  Detail & Related papers  (2023-06-01T09:49:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.