BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation

• URL: http://arxiv.org/abs/2601.15123v1
• Date: Wed, 21 Jan 2026 16:02:21 GMT
• Title: BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation
• Authors: Andrey Moskalenko, Danil Kuznetsov, Irina Dudko, Anastasiia Iasakova, Nikita Boldyrev, Denis Shepelev, Andrei Spiridonov, Andrey Kuznetsov, Vlad Shakhuro,
• Abstract summary: We investigate the robustness of promptable segmentation models to natural variations in bounding box prompts.<n>Our analysis reveals substantial variability in segmentation quality across users for the same model and instance.<n>We introduce BREPS, a method for generating adversarial bounding boxes that minimize or maximize segmentation error.
• Score: 4.5991688539322215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Promptable segmentation models such as SAM have established a powerful paradigm, enabling strong generalization to unseen objects and domains with minimal user input, including points, bounding boxes, and text prompts. Among these, bounding boxes stand out as particularly effective, often outperforming points while significantly reducing annotation costs. However, current training and evaluation protocols typically rely on synthetic prompts generated through simple heuristics, offering limited insight into real-world robustness. In this paper, we investigate the robustness of promptable segmentation models to natural variations in bounding box prompts. First, we conduct a controlled user study and collect thousands of real bounding box annotations. Our analysis reveals substantial variability in segmentation quality across users for the same model and instance, indicating that SAM-like models are highly sensitive to natural prompt noise. Then, since exhaustive testing of all possible user inputs is computationally prohibitive, we reformulate robustness evaluation as a white-box optimization problem over the bounding box prompt space. We introduce BREPS, a method for generating adversarial bounding boxes that minimize or maximize segmentation error while adhering to naturalness constraints. Finally, we benchmark state-of-the-art models across 10 datasets, spanning everyday scenes to medical imaging. Code - https://github.com/emb-ai/BREPS.

Related papers

• On-the-Fly OVD Adaptation with FLAME: Few-shot Localization via Active Marginal-Samples Exploration [1.7975230539002824]
  Open-vocabulary object detection (OVD) models offer remarkable flexibility by detecting objects from arbitrary text queries.<n>Their zero-shot performance in specialized domains like Remote Sensing (RS) is often compromised by the inherent ambiguity of natural language.<n>We propose a cascaded approach that couples the broad generalization of a large pre-trained OVD model with a lightweight few-shot classifier.
  arXiv  Detail & Related papers  (2025-10-20T15:41:55Z)
• Boundary on the Table: Efficient Black-Box Decision-Based Attacks for Structured Data [2.02409171087469]
  Adversarial robustness in structured data remains an underexplored frontier compared to vision and language domains.<n>Our approach combines gradient-free direction estimation with an iterative boundary search, enabling efficient navigation of discrete and continuous feature spaces.<n>Experiments demonstrate that our method successfully compromises nearly the entire test set across diverse models.
  arXiv  Detail & Related papers  (2025-09-26T19:00:11Z)
• Prompt learning with bounding box constraints for medical image segmentation [9.429796437031577]
  Vision foundation models have recently shown noteworthy segmentation performance when provided with prompts such as points or bounding boxes.<n>This paper proposes a novel framework that combines the representational power of foundation models with the annotation efficiency of weakly supervised segmentation.<n>Our method achieves an average Dice score of 84.90% in a limited data setting, outperforming existing fully-supervised and weakly-supervised approaches.
  arXiv  Detail & Related papers  (2025-07-03T16:04:08Z)
• ProSAM: Enhancing the Robustness of SAM-based Visual Reference Segmentation with Probabilistic Prompts [15.582637232358177]
  We introduce ProSAM, a simple but effective method to address the stability challenges we identified in existing SAM-based visual reference segmentation approaches.<n>ProSAM avoids generating prompts that lie in unstable regions, overcoming the instability caused by less robust prompts.<n>Our approach consistently surpasses state-of-the-art methods on the Pascal-5$i$ and COCO-20$i$ datasets.
  arXiv  Detail & Related papers  (2025-06-27T00:50:15Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• Multi-Attribute Constraint Satisfaction via Language Model Rewriting [67.5778646504987]
  Multi-Attribute Constraint Satisfaction (MACS) is a method capable of finetuning language models to satisfy user-specified constraints on multiple external real-value attributes.<n>Our work opens new avenues for generalized and real-value multi-attribute control, with implications for diverse applications spanning NLP and bioinformatics.
  arXiv  Detail & Related papers  (2024-12-26T12:36:39Z)
• Auto-Prompt Generation is Not Robust: Prompt Optimization Driven by Pseudo Gradient [50.15090865963094]
  We introduce PertBench, a comprehensive benchmark dataset that includes a wide range of input perturbations.<n>Our analysis reveals substantial vulnerabilities in existing prompt generation strategies.<n>We propose PGO, a gradient-free prompt generation framework that leverages perturbation types as pseudo-gradient signals.
  arXiv  Detail & Related papers  (2024-12-24T06:05:08Z)
• Hyperband-based Bayesian Optimization for Black-box Prompt Selection [15.756224286651237]
  Black-box prompt selection is challenging due to potentially large, search spaces, absence of gradient information, and high evaluation cost of prompts on a validation set.<n>We propose HbBoPs, a novel method that combines a structural-aware deep kernel Gaussian Process with Hyperband as a multi-fidelity scheduler.<n>HbBoPs outperforms state-of-the-art methods in both performance and efficiency.
  arXiv  Detail & Related papers  (2024-12-10T14:42:51Z)
• ClickTrack: Towards Real-time Interactive Single Object Tracking [58.52366657445601]
  We propose a new paradigm for single object tracking algorithms, ClickTrack, a new paradigm using clicking interaction for real-time scenarios. To address ambiguity in certain special scenarios, we designed the Guided Click Refiner(GCR), which accepts point and optional textual information as inputs. Experiments on LaSOT and GOT-10k benchmarks show that tracker combined with GCR achieves stable performance in real-time interactive scenarios.
  arXiv  Detail & Related papers  (2024-11-20T10:30:33Z)
• Neighbor-Aware Calibration of Segmentation Networks with Penalty-Based Constraints [19.897181782914437]
  We propose a principled and simple solution based on equality constraints on the logit values, which enables to control explicitly both the enforced constraint and the weight of the penalty. Our approach can be used to train a wide span of deep segmentation networks.
  arXiv  Detail & Related papers  (2024-01-25T19:46:57Z)
• Attribute-Guided Adversarial Training for Robustness to Natural Perturbations [64.35805267250682]
  We propose an adversarial training approach which learns to generate new samples so as to maximize exposure of the classifier to the attributes-space. Our approach enables deep neural networks to be robust against a wide range of naturally occurring perturbations.
  arXiv  Detail & Related papers  (2020-12-03T10:17:30Z)
• The Simulator: Understanding Adaptive Sampling in the Moderate-Confidence Regime [52.38455827779212]
  We propose a novel technique for analyzing adaptive sampling called the em Simulator. We prove the first instance-based lower bounds the top-k problem which incorporate the appropriate log-factors. Our new analysis inspires a simple and near-optimal for the best-arm and top-k identification, the first em practical of its kind for the latter problem.
  arXiv  Detail & Related papers  (2017-02-16T23:42:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.