How Universal Are SAM2 Features?

• URL: http://arxiv.org/abs/2510.17051v1
• Date: Sun, 19 Oct 2025 23:31:37 GMT
• Title: How Universal Are SAM2 Features?
• Authors: Masoud Khairi Atani, Alon Harell, Hyomin Choi, Runyu Yang, Fabien Racape, Ivan V. Bajic,
• Abstract summary: We compare the general-purpose Hiera encoder against the segmentation-specialized Segment Anything Model 2 (SAM2)<n>Using a lightweight, trainable neck to probe the adaptability of their frozen features, we quantify the information-theoretic cost of specialization.<n>Our results reveal that while SAM2's specialization is highly effective for spatially-related tasks like depth estimation, it comes at a cost.
• Score: 14.833819368322091
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The trade-off between general-purpose foundation vision models and their specialized counterparts is critical for efficient feature coding design and is not yet fully understood. We investigate this trade-off by comparing the feature versatility of the general-purpose Hiera encoder against the segmentation-specialized Segment Anything Model 2 (SAM2). Using a lightweight, trainable neck to probe the adaptability of their frozen features, we quantify the information-theoretic cost of specialization. Our results reveal that while SAM2's specialization is highly effective for spatially-related tasks like depth estimation, it comes at a cost. The specialized SAM2 encoder underperforms its generalist predecessor, Hiera, on conceptually distant tasks such as pose estimation and image captioning, demonstrating a measurable loss of broader semantic information. A novel cross-neck analysis on SAM2 reveals that each level of adaptation creates a further representational bottleneck. Our analysis illuminates these trade-offs in feature universality, providing a quantitative foundation for designing efficient feature coding and adaptation strategies for diverse downstream applications.

Related papers

• Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder [59.89996751196727]
  Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models.<n>SAEs' hidden layers have high dimensionality to satisfy sparsity constraints, resulting in prohibitive training and inference costs.<n>Recent Mixture of Experts (MoE) approaches attempt to address this by SAEs into narrower expert networks with gated activation.<n>We propose two key innovations: (1) Multiple Expert Activation that simultaneously engages semantically weighted expert subsets to encourage specialization, and (2) Feature Scaling that enhances diversity through adaptive high-frequency scaling.
  arXiv  Detail & Related papers  (2025-11-07T22:19:34Z)
• SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Tasks [50.97089872043121]
  We propose SAM2-UNeXT, an advanced framework that builds upon the core principles of SAM2-UNet.<n>We extend the representational capacity of SAM2 through the integration of an auxiliary DINOv2 encoder.<n>Our approach enables more accurate segmentation with a simple architecture, relaxing the need for complex decoder designs.
  arXiv  Detail & Related papers  (2025-08-05T15:36:13Z)
• O2Former:Direction-Aware and Multi-Scale Query Enhancement for SAR Ship Instance Segmentation [0.3611754783778107]
  Instance segmentation of ships in synthetic aperture radar (SAR) imagery is critical for applications such as maritime monitoring, environmental analysis, and national security.<n> SAR ship images present challenges including scale variation, object density, and fuzzy target boundary.<n>We propose O2Former, a tailored instance segmentation framework that extends Mask2Former by fully leveraging the structural characteristics of SAR imagery.
  arXiv  Detail & Related papers  (2025-06-13T16:06:51Z)
• UrbanSAM: Learning Invariance-Inspired Adapters for Segment Anything Models in Urban Construction [51.54946346023673]
  Urban morphology is inherently complex, with irregular objects of diverse shapes and varying scales.<n>The Segment Anything Model (SAM) has shown significant potential in segmenting complex scenes.<n>We propose UrbanSAM, a customized version of SAM specifically designed to analyze complex urban environments.
  arXiv  Detail & Related papers  (2025-02-21T04:25:19Z)
• Adapting Segment Anything Model for Unseen Object Instance Segmentation [70.60171342436092]
  Unseen Object Instance (UOIS) is crucial for autonomous robots operating in unstructured environments. We propose UOIS-SAM, a data-efficient solution for the UOIS task. UOIS-SAM integrates two key components: (i) a Heatmap-based Prompt Generator (HPG) to generate class-agnostic point prompts with precise foreground prediction, and (ii) a Hierarchical Discrimination Network (HDNet) that adapts SAM's mask decoder.
  arXiv  Detail & Related papers  (2024-09-23T19:05:50Z)
• A SAM-guided Two-stream Lightweight Model for Anomaly Detection [44.73985145110819]
  We propose a SAM-guided Two-stream Lightweight Model for unsupervised anomaly detection (STLM) Our experiments conducted on MVTec AD benchmark show that STLM, with about 16M parameters and achieving an inference time in 20ms, competes effectively with state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-29T13:29:10Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• Multi-scale and Cross-scale Contrastive Learning for Semantic Segmentation [5.281694565226513]
  We apply contrastive learning to enhance the discriminative power of the multi-scale features extracted by semantic segmentation networks. By first mapping the encoder's multi-scale representations to a common feature space, we instantiate a novel form of supervised local-global constraint.
  arXiv  Detail & Related papers  (2022-03-25T01:24:24Z)
• Lightweight Single-Image Super-Resolution Network with Attentive Auxiliary Feature Learning [73.75457731689858]
  We develop a computation efficient yet accurate network based on the proposed attentive auxiliary features (A$2$F) for SISR. Experimental results on large-scale dataset demonstrate the effectiveness of the proposed model against the state-of-the-art (SOTA) SR methods.
  arXiv  Detail & Related papers  (2020-11-13T06:01:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.