Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing

• URL: http://arxiv.org/abs/2607.09583v1
• Date: Fri, 10 Jul 2026 16:33:05 GMT
• Title: Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing
• Abstract summary: We present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation.<n>Our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models.<n>It remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots.
• Score: 0.42970700836450487
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The deployment of large-scale foundation models, such as the Segment Anything Model 3 (SAM 3), promises a transition toward open-vocabulary, training-free computer vision. However, their capacity to generalize out-of-distribution to the complex, top-down geometric structures of Earth Observation imagery remains largely unquantified. Driven by SAM 3's performance disparities in highly specialized domains, we present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation under strict zero-shot and one-shot constraints. To achieve this, we introduce a structural adaptation of SAM 3 by repurposing its decoupled binary presence head into a standalone zero-shot classifier. Furthermore, by systematically isolating textual and visual prompt modalities across five configurations, we explicitly diagnose the alignment mechanics within the model's multimodal decoder. Our findings reveal severe cross-modal interference: while visual prompts successfully align the decoder to complex remote sensing geometry, textual prompts inject misaligned, ground-level semantic bias, actively degrading coordinate regression. To benchmark these capabilities without resource-intensive training, we formulate a novel training-free proxy evaluation protocol for Generalized Zero-Shot tasks (scene classification and instance segmentation). Ultimately, our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models, achieving high Harmonic Mean scores in segmentation tasks. However, it remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots, charting a definitive mandate for parameter-efficient geospatial fine-tuning of its multimodal decoder.

Related papers

• Training-free Cross-domain Few-shot Segmentation via Robust Semantic Representation and Matching [8.277992670183004]
  Cross-domain Few-shot (CD-FSS) aims to transfer knowledge learned from source domain to distinct target domains.<n>We propose a training-free framework to avoid both training overhead and overfitting.
  arXiv  Detail & Related papers  (2026-06-23T08:19:30Z)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• Improving Visual Grounding in Remote Sensing via Cluster-Guided Refinement and Model Ensemble Voting [0.16332728502735253]
  Visual grounding aims to locate image regions that correspond to natural language descriptions.<n>In remote sensing imagery, grounding is particularly challenging due to complex scenes, small objects, and large variations in scale.<n>We propose two grounding pipelines, that combine the complementary strengths of RemoteSAM, a visual grounding model specialized for remote sensing, and SAM3, a powerful general-purpose segmentation model.
  arXiv  Detail & Related papers  (2026-05-30T06:13:42Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere [90.85001795656633]
  Video anomaly detection (VAD) aims to automatically identify events that deviate from normal patterns in untrimmed surveillance videos.<n>We propose SphereVAD, a fully training-free, zero-shot VAD framework that recasts anomaly discrimination as von Mises-Fisher (vMF) likelihood-ratio geodesic inference on the unit hypersphere.
  arXiv  Detail & Related papers  (2026-05-08T16:57:38Z)
• Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement [24.433000105666114]
  We propose a parameter-efficient fine-tuning framework, named Cooperative Fine-Grained Refinement of SAM.<n>These three components work cooperatively to enhance local perception, preserve spatial details, and refine boundaries, enabling SAM to perform accurate nuclei instance segmentation directly.
  arXiv  Detail & Related papers  (2026-03-30T04:39:07Z)
• Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds [53.82500407523346]
  PointINS is an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning.<n>PointINS achieves on average +3.5% mAP improvement for indoor instance segmentation and +4.1% PQ gain for outdoor panoptic segmentation.
  arXiv  Detail & Related papers  (2026-03-26T08:31:06Z)
• SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images [51.42466259821335]
  We present a preliminary exploration of applying SAM 3 to the remote sensing OVSS task without any training.<n>First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder.<n>Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene.
  arXiv  Detail & Related papers  (2025-12-09T15:42:28Z)
• Towards 3D Object-Centric Feature Learning for Semantic Scene Completion [18.41627244498394]
  Vision-based 3D Semantic Scene Completion (SSC) has received growing attention due to its potential in autonomous driving.<n>We propose Ocean, an object-centric prediction framework that decomposes the scene into individual object instances.<n>We show that Ocean achieves state-of-the-art performance, with mIoU scores of 17.40 and 20.28, respectively.
  arXiv  Detail & Related papers  (2025-11-17T06:28:26Z)
• S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Driving [12.406655155106424]
  We propose S3PT a novel scene semantics and structure guided clustering to provide more scene-consistent objectives for self-supervised training.<n>Our contributions are threefold: First, we incorporate semantic distribution consistent clustering to encourage better representation of rare classes such as motorcycles or animals.<n>Second, we introduce object diversity consistent spatial clustering, to handle imbalanced and diverse object sizes, ranging from large background areas to small objects such as pedestrians and traffic signs.<n>Third, we propose a depth-guided spatial clustering to regularize learning based on geometric information of the scene, thus further refining region separation on the feature level.
  arXiv  Detail & Related papers  (2024-10-30T15:00:06Z)
• View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields [52.08335264414515]
  We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene. Our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output. We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency.
  arXiv  Detail & Related papers  (2024-05-30T04:14:58Z)
• Benchmarking Unsupervised Object Representations for Video Sequences [111.81492107649889]
  We compare the perceptual abilities of four object-centric approaches: ViMON, OP3, TBA and SCALOR. Our results suggest that the architectures with unconstrained latent representations learn more powerful representations in terms of object detection, segmentation and tracking. Our benchmark may provide fruitful guidance towards learning more robust object-centric video representations.
  arXiv  Detail & Related papers  (2020-06-12T09:37:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.