RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation

• URL: http://arxiv.org/abs/2605.07334v1
• Date: Fri, 08 May 2026 06:39:53 GMT
• Title: RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentation
• Abstract summary: Video Reasoning aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic.<n>Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM.<n>We introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video (TVR) and target perception (KTP)
• Score: 48.30592530624143
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Reasoning Segmentation (VRS) aims to segment target objects in videos based on implicit instructions that convey human intent and temporal logic. Existing MLLM-based methods predict masks with a [SEG] token after selecting frames via simple sampling or an auxiliary MLLM, where limited supervision and frame-language similarity rules often yield narrow-scope keyframe choices that weaken holistic temporal understanding and lead to brittle localization in complex multi-object scenes. To address these issues, we introduce RCoT-Seg, a video-of-thought framework that factorizes VRS into temporal video reasoning (TVR) and keyframe target perception (KTP), explicitly separating temporal reasoning from spatial perception. Specifically, in the TVR stage, an agentic keyframe selection module, initialized with a curated CoT-start corpus and refined by GRPO under task-aligned rewards, is proposed to generate and reselect the keyframe through self-evaluation, strengthening moment localization and temporal reasoning. In the KTP stage, RCoT-Seg performs high-resolution segmentation on the selected frame and propagates masks with SAM2-based methods across the sequence, replacing heuristic sampling and external selectors while improving spatial precision and inter-frame consistency. Extensive experimental results demonstrate that the proposed RCoT-Seg achieves favorable performance against the state-of-the-art methods. The code and models will be publicly released at https://github.com/Victor-wjw/RCoT-Seg.

Related papers

• Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs [81.04673240949074]
  Reinforcement learning has improved the reasoning ability of large language models.<n>Applying outcome-only rewards to video multimodal large language models provides limited guidance on which visual evidence should support the answer.<n>We introduce Consensus Frame GRPO, a temporal-annotation-free process-level reward framework for evidence-aware video reasoning.
  arXiv  Detail & Related papers  (2026-06-16T19:42:54Z)
• Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understanding [48.14432643308697]
  Long video understanding is a formidable challenge for Multimodal Large Language Models.<n>Q-Gate is a plug-and-play framework that treats selection as a dynamic modality routing problem.
  arXiv  Detail & Related papers  (2026-04-19T13:04:18Z)
• VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation [6.447274127678917]
  Video Video Object (RVOS) aims to segment target objects in videos based on natural language descriptions.<n>We propose VIRST (Video-Instructed Reasoning Assistant for Spatio-Temporal), an end-to-end framework that unifies global video reasoning and pixel-level mask prediction within a single model.
  arXiv  Detail & Related papers  (2026-03-28T00:34:15Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding [40.37010049965347]
  Referring Video Object (RefVOS) seeks to segment target objects in videos guided by natural language descriptions.<n>We propose a unified framework that jointly optimize Temporal Sentence Grounding (TSG) and RefVOS, naturally incorporating key moment grounding capability.
  arXiv  Detail & Related papers  (2025-10-10T11:18:21Z)
• CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos [59.391265901911005]
  We propose CoT-RVS, a novel framework employing the zero-shot Chain-of-Thought (CoT) capability of MLLM to address complex challenges by temporal-semantic reasoning.<n>CoT-RVS analyzes the visible objects within a given frame that possibly match the language query (semantic), and chooses a corresponding for each object that can be observed effortlessly among all frames (temporal)<n>Our framework's training-free feature further allows its extension to process online video streams, where the CoT is used at test time to update the object of interest when a better target starts to emerge
  arXiv  Detail & Related papers  (2025-05-24T07:01:31Z)
• The Devil is in Temporal Token: High Quality Video Reasoning Segmentation [68.33080352141653]
  Methods for Video Reasoning rely heavily on a single special token to represent the object in the video.<n>We propose VRS-HQ, an end-to-end video reasoning segmentation approach.<n>Our results highlight the strong temporal reasoning and segmentation capabilities of our method.
  arXiv  Detail & Related papers  (2025-01-15T03:17:24Z)
• Temporally Consistent Referring Video Object Segmentation with Hybrid Memory [98.80249255577304]
  We propose an end-to-end R-VOS paradigm that explicitly models temporal consistency alongside the referring segmentation. Features of frames with automatically generated high-quality reference masks are propagated to segment remaining frames. Extensive experiments demonstrate that our approach enhances temporal consistency by a significant margin.
  arXiv  Detail & Related papers  (2024-03-28T13:32:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.