Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search

• URL: http://arxiv.org/abs/2602.04454v1
• Date: Wed, 04 Feb 2026 11:33:16 GMT
• Title: Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
• Abstract summary: Seg-ReSearch is a novel segmentation paradigm that overcomes the knowledge bottleneck of existing approaches.<n>By enabling interleaved reasoning and external search, Seg-ReSearch empowers segmentation systems to handle dynamic, open-world queries.<n> Experiments on OK-VOS and two existing reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin.
• Score: 45.79431884815933
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Segmentation based on language has been a popular topic in computer vision. While recent advances in multimodal large language models (MLLMs) have endowed segmentation systems with reasoning capabilities, these efforts remain confined by the frozen internal knowledge of MLLMs, which limits their potential for real-world scenarios that involve up-to-date information or domain-specific concepts. In this work, we propose \textbf{Seg-ReSearch}, a novel segmentation paradigm that overcomes the knowledge bottleneck of existing approaches. By enabling interleaved reasoning and external search, Seg-ReSearch empowers segmentation systems to handle dynamic, open-world queries that extend beyond the frozen knowledge of MLLMs. To effectively train this capability, we introduce a hierarchical reward design that harmonizes initial guidance with progressive incentives, mitigating the dilemma between sparse outcome signals and rigid step-wise supervision. For evaluation, we construct OK-VOS, a challenging benchmark that explicitly requires outside knowledge for video object segmentation. Experiments on OK-VOS and two existing reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin. Code and data will be released at https://github.com/iSEE-Laboratory/Seg-ReSearch.

Related papers

• Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification [21.7465671470498]
  Referring segmentation aims to segment the target objects in images or videos based on the textual query.<n>We propose IC-Seg, a novel agentic framework that proactively clarifies user intent through multi-turn conversation.<n>We also introduce Hi-GRPO, a new hierarchical optimization strategy that injects dense and informative supervision signals at the trajectory, turn, and step levels.
  arXiv  Detail & Related papers  (2026-05-17T16:30:44Z)
• Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation [52.8308168727975]
  Seg-Agent is a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning.<n>Our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement.<n>Various-LangSeg is a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks.
  arXiv  Detail & Related papers  (2026-05-13T03:36:44Z)
• ROSE: Retrieval-Oriented Segmentation Enhancement [79.7370065070812]
  Existing segmentation models based on multimodal large language models (MLLMs) often struggle with novel or emerging entities due to their inability to incorporate up-to-date knowledge.<n>We introduce the Novel Emerging Task (NEST), which focuses on segmenting (i) novel entities that MLLMs fail to recognize due to their absence from training data, and (ii) emerging entities that exist within the model's knowledge but demand up-to-date information for accurate recognition.<n>Rose: Retrieval-Oriented Enhancement, a plug-and-play framework designed to augment any MLLM-based segmentation model,
  arXiv  Detail & Related papers  (2026-04-15T17:59:35Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• Contrastive Concept-Tree Search for LLM-Assisted Algorithm Discovery [10.823958143531685]
  Large language Model (LLM)-assisted algorithm discovery is an iterative, black-box optimization process over programs.<n>Here, we introduce Contrastive Concept-Tree Search (CCTS), which extracts a hierarchical concept representation from the generated programs.<n>We show that CCTS improves search efficiency over fitness-based baselines and produces interpretable, task-specific concept trees.
  arXiv  Detail & Related papers  (2026-02-03T05:41:35Z)
• Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Remote Sensing [8.731693840957716]
  Think2Seg-RS is a framework that trains an LVLM prompter to control a frozen Segment Anything Model (SAM) via structured geometric prompts.<n>The framework achieves state-of-the-art performance on the EarthReason dataset.<n> compact segmenters outperform larger ones under semantic-level supervision, and that negative prompts are ineffective in heterogeneous aerial backgrounds.
  arXiv  Detail & Related papers  (2025-12-22T11:46:42Z)
• FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning [62.11389260206383]
  textscFineRS is a two-stage MLLM-based reinforcement learning framework for segmenting extremely small objects.<n>We present textscFineRS-4k, a new dataset for evaluating MLLMs on attribute-level reasoning and pixel-level segmentation on subtle, small-scale targets.
  arXiv  Detail & Related papers  (2025-10-24T10:14:17Z)
• Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughts [64.93416171745693]
  ThinkFirst is a training-free reasoning segmentation framework.<n>Our approach allows GPT-4o or other powerful MLLMs to generate a detailed, chain-of-thought description of an image.<n>This summarized description is then passed to a language-instructed segmentation assistant to aid the segmentation process.
  arXiv  Detail & Related papers  (2025-03-10T16:26:11Z)
• Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generation [0.0]
  LangSeg is a novel semantic segmentation method that leverages context-sensitive, fine-grained subclass descriptors.<n>We evaluate LangSeg on two challenging datasets, ADE20K and COCO-Stuff, where it outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2025-01-27T20:02:12Z)
• Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Prompting [23.61061000692023]
  This study proposes leveraging user interactions recorded in search logs to yield insights into users' implicit search intentions.<n>We propose ProRBP, a novel Progressive Retrieved Behavior-augmented Prompting framework for integrating search scenario-oriented knowledge with Large Language Models.
  arXiv  Detail & Related papers  (2024-08-18T11:07:38Z)
• Frequency-based Matcher for Long-tailed Semantic Segmentation [22.199174076366003]
  We focus on a relatively under-explored task setting, long-tailed semantic segmentation (LTSS) We propose a dual-metric evaluation system and construct the LTSS benchmark to demonstrate the performance of semantic segmentation methods and long-tailed solutions. We also propose a transformer-based algorithm to improve LTSS, frequency-based matcher, which solves the oversuppression problem by one-to-many matching.
  arXiv  Detail & Related papers  (2024-06-06T09:57:56Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• LISA: Reasoning Segmentation via Large Language Model [68.24075852136761]
  We propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. We present LISA: large Language Instructed Assistant, which inherits the language generation capabilities of multimodal Large Language Models.
  arXiv  Detail & Related papers  (2023-08-01T17:50:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.