Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation

• URL: http://arxiv.org/abs/2602.06333v1
• Date: Fri, 06 Feb 2026 02:59:11 GMT
• Title: Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation
• Abstract summary: We present textscConceptBank, a framework to restore alignment between visual evidence and prompts on the fly.<n>Our approach anchors target-domain evidence via class-wise visual prototypes, (textitii) mines representative supports to suppress outliers under data drift, and (textitiii) fuses candidate concepts to rectify concept drift.
• Score: 57.427604620940734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent introduction of \texttt{SAM3} has revolutionized Open-Vocabulary Segmentation (OVS) through \textit{promptable concept segmentation}, which grounds pixel predictions in flexible concept prompts. However, this reliance on pre-defined concepts makes the model vulnerable: when visual distributions shift (\textit{data drift}) or conditional label distributions evolve (\textit{concept drift}) in the target domain, the alignment between visual evidence and prompts breaks down. In this work, we present \textsc{ConceptBank}, a parameter-free calibration framework to restore this alignment on the fly. Instead of adhering to static prompts, we construct a dataset-specific concept bank from the target statistics. Our approach (\textit{i}) anchors target-domain evidence via class-wise visual prototypes, (\textit{ii}) mines representative supports to suppress outliers under data drift, and (\textit{iii}) fuses candidate concepts to rectify concept drift. We demonstrate that \textsc{ConceptBank} effectively adapts \texttt{SAM3} to distribution drifts, including challenging natural-scene and remote-sensing scenarios, establishing a new baseline for robustness and efficiency in OVS. Code and model are available at https://github.com/pgsmall/ConceptBank.

Related papers

• Interpreting Object-Dependent Concept Brittleness in Text-to-Image Diffusion Models [105.53190946111586]
  We term this phenomenon object-dependent concept brittleness.<n>We present an interpretability-oriented framework to audit and minimally correct these failures.<n>We evaluate the proposed framework on style and attribute failure cases across multiple diffusion backbones.
  arXiv  Detail & Related papers  (2026-09-09T09:07:40Z)
• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• GFR-SAM: Training-Free Referring Camouflaged Object Segmentation via Cross-Image Prompting [40.66340261994875]
  Referring Camouflaged Object Detection (Ref-COD) requires segmenting hidden targets guided by reference cues.<n>We propose GFR-SAM, a robust three-stage training-free framework.<n>First, we introduce In-Context Exemplar-guided exemplars, empowering SAM3 with cross-image inference to generate candidate masks.<n>Second, a Region-Global Contrastive Filtering module ranks candidates through DINOv3-based prototypical alignment.<n>Third, a Geometric-Semantic Refinement module synergizes bounding box and text prompts to recover fine-grained boundaries and enhance instance recall.
  arXiv  Detail & Related papers  (2026-07-13T15:55:45Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• Beyond Distribution Estimation: Simplex Anchored Structural Inference Towards Universal Semi-Supervised Learning [47.92504362652887]
  Semi-supervised learning faces significant challenges in realistic scenarios where labeled data is scarce and unlabeled data follows unknown, arbitrary distributions.<n>We formalize this critical yet under-explored paradigm as Universal Semi-supervised Learning (UniSSL)<n>We introduce a weighting strategy based on distribution-agnostic metrics to prioritize reliable pseudo-labels and an auxiliary branch to isolate potentially erroneous pseudo-labels.
  arXiv  Detail & Related papers  (2026-05-08T10:33:09Z)
• Few-Shot Semantic Segmentation Meets SAM3 [23.28277367328234]
  Few-Shot Semantic (FSS) focuses on segmenting novel object categories from only a handful of annotated examples.<n>We revisit FSS from the perspective of modern vision foundation models and explore the potential of Segment Anything Model 3 (SAM3) as a training-free solution.<n>By repurposing its Promptable Concept (PCS) capability, we adopt a simple spatial concatenation strategy that places support and query images into a shared canvas.
  arXiv  Detail & Related papers  (2026-04-07T04:59:50Z)
• Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition [81.2779530670268]
  Vision-Language Models (VLMs) have significantly advanced zero-shot image recognition.<n>This paper enhances prompt by incorporating class-specific concepts.<n>Our method consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-03-09T03:11:11Z)
• StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models [57.674757786328236]
  StructSAM is a resolution-preserving merge-unmerge framework tailored to Segment Anything Model (SAM)<n>We show that StructSAM reduces encoder FLOPs by 25-30% (up to 40%+ with prompt-aware merging) with minor drops in mIoU/Dice.<n>We also provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines.
  arXiv  Detail & Related papers  (2026-03-07T18:30:58Z)
• Invariance on Manifolds: Understanding Robust Visual Representations for Place Recognition [19.200074425090595]
  We propose a Second-Order Geometric Statistics framework that inherently captures geometric stability without training.<n>Our approach introduces a training-free framework built upon fixed, pre-trained backbones, achieving strong zero-shot generalization without parameter updates.
  arXiv  Detail & Related papers  (2026-01-31T18:12:29Z)
• Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion [31.189038928192648]
  Co2S is a semi-supervised RS segmentation framework that fuses priors from vision-language models and self-supervised models.<n>An explicit-implicit semantic co-guidance mechanism is introduced that utilizes text embeddings and learnable queries.<n>Experiments on six popular datasets demonstrate the superiority of the proposed method.
  arXiv  Detail & Related papers  (2025-12-28T18:24:19Z)
• From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models [81.43473418572567]
  Click-Through Rate (CTR) prediction is a core task in recommendation systems.<n>We propose a novel generative framework to address embedding dimensional collapse and information redundancy.<n>We show that SFG consistently mitigates embedding collapse and reduces information redundancy, while yielding substantial performance gains.
  arXiv  Detail & Related papers  (2025-12-16T03:17:18Z)
• TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection [62.95726973851089]
  TokenCLIP is a token-wise adaptation framework for anomaly learning.<n>It enables dynamic alignment between visual and learnable textual spaces for fine-grained anomaly learning.
  arXiv  Detail & Related papers  (2025-10-24T05:51:31Z)
• Personalized OVSS: Understanding Personal Concept in Open-Vocabulary Semantic Segmentation [59.047277629795325]
  We introduce a novel task termed textitpersonalized open-vocabulary semantic segmentation'<n>We propose a text prompt tuning-based plug-in method designed to recognize personal visual concepts using a few pairs of images and masks.<n>We further improve the performance by enriching the representation of text prompts by injecting visual embeddings of the personal concept into them.
  arXiv  Detail & Related papers  (2025-07-15T06:51:07Z)
• Adaptive Concept Bottleneck for Foundation Models Under Distribution Shifts [33.677249894085186]
  We explore the potential of Concept Bottleneck Models for transforming complex, non-interpretable foundation models into interpretable decision-making pipelines.<n>Specifically, we focus on the test-time deployment of such an interpretable CBM pipeline "in the wild"<n>Our adaptation method produces concept-based interpretations better aligned with the test data and boosts post-deployment accuracy by up to 28%.
  arXiv  Detail & Related papers  (2024-12-18T17:47:46Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery [52.498055901649025]
  Concept Bottleneck Models (CBMs) have been proposed to address the 'black-box' problem of deep neural networks. We propose a novel CBM approach -- called Discover-then-Name-CBM (DN-CBM) -- that inverts the typical paradigm. Our concept extraction strategy is efficient, since it is agnostic to the downstream task, and uses concepts already known to the model.
  arXiv  Detail & Related papers  (2024-07-19T17:50:11Z)
• Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Models [14.899741072838994]
  Recent CLIP-like Vision-Language Models (VLMs), pre-trained on large amounts of image-text pairs, have paved the way to open-vocabulary semantic segmentation.<n>We propose two different approaches to automatically generate, at test time, query-specific textual contrastive concepts.
  arXiv  Detail & Related papers  (2024-07-06T12:18:43Z)
• Neural Concept Binder [22.074896812195437]
  We introduce the Neural Concept Binder (NCB), a framework for deriving both discrete and continuous concept representations. The structured nature of NCB's concept representations allows for intuitive inspection and the straightforward integration of external knowledge. We validate the effectiveness of NCB through evaluations on our newly introduced CLEVR-Sudoku dataset.
  arXiv  Detail & Related papers  (2024-06-14T11:52:09Z)
• Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency [2.7719338074999547]
  Concept bottleneck models (CBMs) have emerged as critical tools in domains where interpretability is paramount. This study proposes underlinetextbfConceptual underlinetextbfLbedding via underlinetextbfEmbedding underlinetextbfApproximations for underlinetextbfReinforcing Interpretability and Transparency.
  arXiv  Detail & Related papers  (2024-06-13T06:04:34Z)
• SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning [49.94607673097326]
  We propose a highly adaptable framework, designated as SimPro, which does not rely on any predefined assumptions about the distribution of unlabeled data. Our framework, grounded in a probabilistic model, innovatively refines the expectation-maximization algorithm. Our method showcases consistent state-of-the-art performance across diverse benchmarks and data distribution scenarios.
  arXiv  Detail & Related papers  (2024-02-21T03:39:04Z)
• Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features? [0.6401548653313325]
  Concept Bottleneck Models (CBMs) are regarded as inherently interpretable because they first predict a set of human-defined concepts. Current literature suggests that concept predictions often rely on irrelevant input features. In this paper, we demonstrate that CBMs can learn to map concepts to semantically meaningful input features.
  arXiv  Detail & Related papers  (2024-02-01T10:18:43Z)
• Advancing Visual Grounding with Scene Knowledge: Benchmark and Method [74.72663425217522]
  Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Most existing VG datasets are constructed using simple description texts. We propose a novel benchmark of underlineScene underlineKnowledge-guided underlineVisual underlineGrounding.
  arXiv  Detail & Related papers  (2023-07-21T13:06:02Z)
• Grounded Object Centric Learning [46.091323528165205]
  We present emphtextscConditional textscSlot textscAttention (textscCoSA) using a novel concept of emphGrounded Slot Dictionary (GSD) inspired by vector quantization. We demonstrate the benefits of our method in multiple downstream tasks such as scene generation, composition, and task adaptation.
  arXiv  Detail & Related papers  (2023-07-18T17:11:55Z)
• Text-To-Concept (and Back) via Cross-Model Alignment [48.133333356834186]
  We show that mapping between an image's representation in one model to its representation in another can be learned surprisingly well with just a linear layer. We convert fixed off-the-shelf vision encoders to surprisingly strong zero-shot classifiers for free. We show other immediate use-cases of text-to-concept, like building concept bottleneck models with no concept supervision.
  arXiv  Detail & Related papers  (2023-05-10T18:01:06Z)
• ContrastMotion: Self-supervised Scene Motion Learning for Large-Scale LiDAR Point Clouds [21.6511040107249]
  We propose a novel self-supervised motion estimator for LiDAR-based autonomous driving via BEV representation. We predict scene motion via feature-level consistency between pillars in consecutive frames, which can eliminate the effect caused by noise points and view-changing point clouds in dynamic scenes.
  arXiv  Detail & Related papers  (2023-04-25T05:46:24Z)
• CPPF++: Uncertainty-Aware Sim2Real Object Pose Estimation by Vote Aggregation [67.12857074801731]
  We introduce a novel method, CPPF++, designed for sim-to-real pose estimation. To address the challenge posed by vote collision, we propose a novel approach that involves modeling the voting uncertainty. We incorporate several innovative modules, including noisy pair filtering, online alignment optimization, and a feature ensemble.
  arXiv  Detail & Related papers  (2022-11-24T03:27:00Z)
• Semantic keypoint-based pose estimation from single RGB frames [64.80395521735463]
  We present an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. We show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios.
  arXiv  Detail & Related papers  (2022-04-12T15:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.