Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification

• URL: http://arxiv.org/abs/2605.17531v2
• Date: Sun, 24 May 2026 12:33:07 GMT
• Title: Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
• Abstract summary: Referring segmentation aims to segment the target objects in images or videos based on the textual query.<n>We propose IC-Seg, a novel agentic framework that proactively clarifies user intent through multi-turn conversation.<n>We also introduce Hi-GRPO, a new hierarchical optimization strategy that injects dense and informative supervision signals at the trajectory, turn, and step levels.
• Score: 21.7465671470498
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring segmentation aims to segment the target objects in images or videos based on the textual query. Despite remarkable progress over the past years, existing works always assume that the user-provided queries are already precise and clear. However, this assumption is impractical. In real-world scenarios, it is unrealistic to expect all users to thoroughly review their visual content and carefully ensure their queries are unique and unambiguous. When encountering such cases, existing segmentation models tend to arbitrarily guess the user preferences, often resulting in undesired outcomes. To address this limitation, we propose IC-Seg, a novel agentic framework that proactively clarifies user intent through multi-turn conversation before segmentation. To effectively incentivize this capability, we further introduce Hi-GRPO, a new hierarchical optimization strategy that injects dense and informative supervision signals at the trajectory, turn, and step levels. This strategy encourages efficient intent clarification, effectively eliminating redundant interactions and improving overall dialogue quality. For evaluation, we establish Ambi-RVOS, a referring video object segmentation benchmark with ambiguous user queries. Extensive experiments demonstrate that IC-Seg not only outperforms existing methods by a large margin in resolving ambiguous queries, but also maintains state-of-the-art performance on standard reasoning segmentation benchmarks. Code and data will be released at https://github.com/iSEE-Laboratory/IC-Seg.

Related papers

• Toward Robust In-Context Segmentation via Concept Guidance [72.30282710763969]
  In-context segmentation (ICS) requires a model to segment target regions in a query image using only a few reference images and their corresponding masks, without updating any parameters.<n>We introduce Concept-Guided In-Context (CG-ICS), which performs segmentation by extracting high-level semantic concepts from references.<n>Experiments on standard ICS benchmarks demonstrate that CG-ICS achieves state-of-the-art accuracy and substantially improves robustness, yielding a more reliable ICS system with significantly reduced variance across diverse reference choices.
  arXiv  Detail & Related papers  (2026-06-26T14:46:38Z)
• Prompting Diffusion Models for Zero-Shot Instance Segmentation [74.37083707535778]
  We present Prompt2Seg, a spatial conditioning framework for diffusion-based segmentation.<n>Our approach takes spatial prompts, represented as 2D Gaussians or confidence maps, as explicit input signals, training the model to respond directly to user intent.<n>We evaluate on seven datasets ranging from standard benchmarks to more challenging domains, including paintings, egocentric views, and X-ray data.
  arXiv  Detail & Related papers  (2026-06-21T20:30:41Z)
• Beyond Referring Expressions: Scenario Comprehension Visual Grounding [18.49037321302958]
  Existing benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category.<n>We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and context rather than explicit naming.<n>We introduce Referring Scenario (RSC), a benchmark designed for this setting. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories.
  arXiv  Detail & Related papers  (2026-04-02T17:59:08Z)
• Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search [45.79431884815933]
  Seg-ReSearch is a novel segmentation paradigm that overcomes the knowledge bottleneck of existing approaches.<n>By enabling interleaved reasoning and external search, Seg-ReSearch empowers segmentation systems to handle dynamic, open-world queries.<n> Experiments on OK-VOS and two existing reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin.
  arXiv  Detail & Related papers  (2026-02-04T11:33:16Z)
• Segment Any Events with Language [68.05185562243356]
  We introduce SEAL, the first Semantic-aware Any Events framework that addresses Open-Vocabulary Event Instance (OV-EIS)<n>Given the visual prompt, our model presents a unified framework to support both segmentation event and open-vocabulary mask classification at multiple levels of granularity, including instance-level and part-level.<n>Our SEAL largely outperforms proposed baselines in terms of performance and inference speed with a parameter-efficient architecture.
  arXiv  Detail & Related papers  (2026-01-30T16:42:56Z)
• SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space [11.534994345027362]
  Multimodal large language models (MLLMs) have shown impressive capabilities in vision-language tasks such as reasoning segmentation.<n>We introduce a novel adversarial paraphrasing task: generating grammatically correct paraphrases that preserve the original query meaning while degrading segmentation performance.<n>We introduce SPARTA-a black-box, sentence-level optimization method that operates in the low-dimensional semantic latent space of a text autoencoder.
  arXiv  Detail & Related papers  (2025-10-28T14:09:05Z)
• SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation [58.80001825332851]
  Referring Image (RIS) aims to segment the target object in an image given a natural language expression.<n>Recent methods predominantly focus on simple expressions like "red car" or "left girl"
  arXiv  Detail & Related papers  (2025-10-11T10:50:58Z)
• Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation [61.37076111486196]
  Ref-AVS aims to segment target objects in audible videos based on given reference expressions.<n>We propose TGS-Agent, which decomposes the task into a Think-Ground-Segment process.<n>Ref-Thinker is a multimodal language model capable of reasoning over textual, visual, and auditory cues.
  arXiv  Detail & Related papers  (2025-08-06T13:05:09Z)
• ConText: Driving In-context Learning for Text Removal and Segmentation [59.6299939669307]
  This paper presents the first study on adapting the visual in-context learning paradigm to optical character recognition tasks.<n>We propose a task-chaining compositor in the form of image-removal-segmentation.<n>We also introduce context-aware aggregation, integrating the chained prompt pattern into the latent query representation.
  arXiv  Detail & Related papers  (2025-06-04T10:06:32Z)
• CLEAR-KGQA: Clarification-Enhanced Ambiguity Resolution for Knowledge Graph Question Answering [13.624962763072899]
  KGQA systems typically assume user queries are unambiguous, which is an assumption that rarely holds in real-world applications.<n>We propose a novel framework that dynamically handles both entity ambiguity (e.g., distinguishing between entities with similar names) and intent ambiguity (e.g., clarifying different interpretations of user queries) through interactive clarification.
  arXiv  Detail & Related papers  (2025-04-13T17:34:35Z)
• Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughts [64.93416171745693]
  ThinkFirst is a training-free reasoning segmentation framework.<n>Our approach allows GPT-4o or other powerful MLLMs to generate a detailed, chain-of-thought description of an image.<n>This summarized description is then passed to a language-instructed segmentation assistant to aid the segmentation process.
  arXiv  Detail & Related papers  (2025-03-10T16:26:11Z)
• Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generation [0.0]
  LangSeg is a novel semantic segmentation method that leverages context-sensitive, fine-grained subclass descriptors.<n>We evaluate LangSeg on two challenging datasets, ADE20K and COCO-Stuff, where it outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2025-01-27T20:02:12Z)
• Learning Spatial-Semantic Features for Robust Video Object Segmentation [108.045326229865]
  We propose a robust video object segmentation framework that learns spatial-semantic features and discriminative object queries.<n>The proposed method achieves state-of-the-art performance on benchmark data sets, including the DAVIS 2017 test (textbf87.8%), YoutubeVOS 2019 (textbf88.1%), MOSE val (textbf74.0%), and LVOS test (textbf73.0%)
  arXiv  Detail & Related papers  (2024-07-10T15:36:00Z)
• SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation [87.18373801829314]
  In-context segmentation aims at segmenting novel images using a few labeled example images, termed as "in-context examples" We propose SEGIC, an end-to-end segment-in-context framework built upon a single vision foundation model (VFM) SEGIC is a straightforward yet effective approach that yields state-of-the-art performance on one-shot segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-11-24T18:59:42Z)
• Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation [79.6596425920849]
  This paper addresses the task of unsupervised video multi-object segmentation. We introduce a novel approach for more accurate and efficient unseen-temporal segmentation. We evaluate the proposed approach on DAVIS$_17$ and YouTube-VIS, and the results demonstrate that it outperforms state-of-the-art methods both in segmentation accuracy and inference speed.
  arXiv  Detail & Related papers  (2021-04-10T14:39:44Z)
• Improving Semantic Segmentation via Decoupled Body and Edge Supervision [89.57847958016981]
  Existing semantic segmentation approaches either aim to improve the object's inner consistency by modeling the global context, or refine objects detail along their boundaries by multi-scale feature fusion. In this paper, a new paradigm for semantic segmentation is proposed. Our insight is that appealing performance of semantic segmentation requires textitexplicitly modeling the object textitbody and textitedge, which correspond to the high and low frequency of the image. We show that the proposed framework with various baselines or backbone networks leads to better object inner consistency and object boundaries.
  arXiv  Detail & Related papers  (2020-07-20T12:11:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.