Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning

• URL: http://arxiv.org/abs/2606.09303v1
• Date: Mon, 08 Jun 2026 10:10:55 GMT
• Title: Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning
• Abstract summary: We propose a two-stage framework Rea2Seg for mask generation and selection.<n>The framework first identifies potential regions as candidate masks based on the attention maps of a segmentation MLLM.<n>It then employs an MLLM to reason over the question and candidate masks and assign scores to each mask.<n>The final segmentation result is obtained by reranking the candidates and selecting the highest-scoring mask.
• Score: 10.180485222685492
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid development of pretrained foundation models has enabled more general image segmentation. Multimodal large language models (MLLMs) have been widely explored for image segmentation with complex queries that require high-level reasoning. Despite promising progress, existing methods are often constrained by limited training data and the gap between MLLMs and mask generation modules. To better transfer MLLMs' perception and reasoning ability to complex reasoning-based segmentation tasks, we propose a two-stage framework Rea2Seg for mask generation and selection. Specifically, the framework first identifies potential regions as candidate masks based on the attention maps of a segmentation MLLM. It then employs an MLLM to reason over the question and candidate masks and assign scores to each mask. The final segmentation result is obtained by reranking the candidates and selecting the highest-scoring mask, reformulating image segmentation as candidate discovery followed by discriminative mask selection. We also notice that a large portion of questions in existing benchmarks focus on commonsense reasoning, and these questions usually do not fully require joint visual observation and reasoning. To address this issue, we introduce a new benchmark called ReasonSeg-SGDR that comprehensively evaluates a model's perception, grounding, and reasoning abilities across multiple dimensions, including discriminative recognition, spatial reasoning, geometric reasoning, and multi-step reasoning, with fine-grained mask generation. In addition, we collect training data to enhance MLLMs' ability to jointly understand multimodal queries and candidate masks, and to assign scores through reasoning. Experimental results on the proposed benchmark and ReasonSeg demonstrate the effectiveness of the unified mask generation and selection framework.

Related papers

• Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning [26.2787717786409]
  We propose textbfSegmentation before textbfAnswering (SegAnswer)<n>SegAnswer shifts the unit of zoom-in from the popular bounding box to pixel-level segmentation mask.<n>It achieves consistent improvements and also exhibits considerable performance on segmentation tasks.
  arXiv  Detail & Related papers  (2026-07-07T03:50:43Z)
• Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation [52.8308168727975]
  Seg-Agent is a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning.<n>Our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement.<n>Various-LangSeg is a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks.
  arXiv  Detail & Related papers  (2026-05-13T03:36:44Z)
• Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation [79.13636675697096]
  Mask Quality Assessment in the Ref-AVS context (MQA-RefAVS)<n>MQA-RefAVS is a task that evaluates the quality of candidate segmentation masks without relying on ground-truth annotations.<n>We propose MQ-Auditor, a multimodal large language model (MLLM)-based auditor that explicitly reasons over multimodal cues and mask information.
  arXiv  Detail & Related papers  (2026-02-03T07:47:59Z)
• Segment and Matte Anything in a Unified Model [5.8874968768571625]
  Segment Anything (SAM) has recently pushed the boundaries of segmentation by demonstrating zero-shot generalization and flexible prompting.<n>We introduce Segment And Matte Anything (SAMA), a lightweight extension of SAM that delivers high-quality interactive image segmentation and matting.
  arXiv  Detail & Related papers  (2026-01-17T19:43:10Z)
• IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation [44.89730606641666]
  IBISAgent reformulates segmentation as a vision-centric, multi-step decision-making process.<n>IBISAgent consistently outperforms both closed-source and open-source SOTA methods.<n>All datasets, code, and trained models will be released publicly.
  arXiv  Detail & Related papers  (2026-01-06T14:37:50Z)
• Refer to Any Segmentation Mask Group With Vision-Language Prompts [79.43440775648824]
  "Refer to Any Mask Group" (RAS) augments segmentation models with complex multimodal interactions and comprehension.<n>We demonstrate superior performance of RAS on our new ORES task, as well as classic referring expression segmentation (RES) and generalized referring expression segmentation (GRES) tasks.
  arXiv  Detail & Related papers  (2025-06-05T17:59:51Z)
• GSVA: Generalized Segmentation via Multimodal Large Language Models [72.57095903188922]
  Generalized Referring Expression (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. Current solutions to GRES remain unsatisfactory since segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt. We propose Generalized Vision Assistant (GSVA) to address this gap.
  arXiv  Detail & Related papers  (2023-12-15T02:54:31Z)
• LISA: Reasoning Segmentation via Large Language Model [68.24075852136761]
  We propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. We present LISA: large Language Instructed Assistant, which inherits the language generation capabilities of multimodal Large Language Models.
  arXiv  Detail & Related papers  (2023-08-01T17:50:17Z)
• MMNet: Multi-Mask Network for Referring Image Segmentation [6.462622145673872]
  We propose an end-to-end Multi-Mask Network for referring image segmentation(MMNet) We first combine picture and language then employ an attention mechanism to generate multiple queries that represent different aspects of the language expression. The final result is obtained through the weighted sum of all masks, which greatly reduces the randomness of the language expression.
  arXiv  Detail & Related papers  (2023-05-24T10:02:27Z)
• Mask Matching Transformer for Few-Shot Segmentation [71.32725963630837]
  Mask Matching Transformer (MM-Former) is a new paradigm for the few-shot segmentation task. First, the MM-Former follows the paradigm of decompose first and then blend, allowing our method to benefit from the advanced potential objects segmenter. We conduct extensive experiments on the popular COCO-$20i$ and Pascal-$5i$ benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T11:00:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.