CoT-Edit: Let CoT Guide Instruction Video Editing
- URL: http://arxiv.org/abs/2608.01113v1
- Date: Sun, 02 Aug 2026 09:20:15 GMT
- Title: CoT-Edit: Let CoT Guide Instruction Video Editing
- Abstract summary: We propose a plan--guide--edit framework that bridges semantic intent and spatial execution.<n>In our framework, a Chain-of-Thought (CoT)-enhanced multimodal large language model (MLLM) serves as a planner.<n>These spatial priors then guide a box-conditioned mask generator, transforming ambiguous global retrieval into localized, context-aware refinement.
- Score: 13.600094788792633
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Text-driven instruction-based video editing in complex scenes remains challenging: purely textual prompts often fail to capture precise spatial relationships and physical constraints, resulting in target ambiguity and physically implausible outcomes. To address this, we propose a plan--guide--edit framework that explicitly bridges semantic intent and spatial execution. In our framework, a Chain-of-Thought (CoT)-enhanced multimodal large language model (MLLM) serves as a planner, performing structured reasoning over the video and instructions to derive a precise sequence of bounding boxes and attribute-enriched editing directives. These spatial priors then guide a box-conditioned mask generator, transforming ambiguous global retrieval into localized, context-aware refinement and producing masks that more accurately capture object scale, contact relationships, and placement. Building on these spatial and semantic signals, a diffusion-based editor integrates the masks, enriched instructions, and frame features to render high-fidelity edits that remain temporally coherent and spatially well aligned. Trained first in a modular manner and then jointly, our framework achieves superior performance with reduced data requirements, delivering precise localization in scenes with multiple similar objects and physically consistent object additions, and extensive experiments demonstrate state-of-the-art performance over multiple strong baseline methods. More details are available at: https://github.com/flying-sky999/CoT-Edit
Related papers
- Text-Vision Co-Instructed Image Editing [11.968553566506626]
We present Text-Vision Co-Instructed Image Editing, which models textual instructions as semantic intent and sparse visual instructions as spatial guidance.<n>By integrating semantic intent and spatial constraints, TV-Edit leads to more precise spatial control, less instruction ambiguity, and stronger structural consistency than text-only or drag-based alternatives.
arXiv Detail & Related papers (2026-06-15T14:16:05Z) - KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing [24.315720861063397]
KGEdit is a structured semantic control framework for text-to-video (T2V) diffusion models.<n>We first construct an ambiguity-aware knowledge graph (AAKG) to disentangle and disambiguate the input prompt.<n>We then design a structured semantic injection module (SSIM) to inject semantic signals into key layers of the diffusion Transformer.
arXiv Detail & Related papers (2026-05-28T07:31:22Z) - Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing [55.211537893248675]
We propose RVEDiT, an implicit Reasoning Video Editing DiT framework built around two complementary components.<n> RVEDiT consistently outperforms state-of-the-art baselines, with particularly strong gains on localized and compositional edits.
arXiv Detail & Related papers (2026-05-23T17:22:14Z) - CEI-3D: Collaborative Explicit-Implicit 3D Reconstruction for Realistic and Fine-Grained Object Editing [51.73433734209541]
Existing 3D editing methods often produce unrealistic and unrefined results due to the deeply integrated nature of their reconstruction networks.<n>This paper introduces CEI-3D, an editing-oriented reconstruction pipeline designed to facilitate realistic and fine-grained editing.<n>Our approach achieves more realistic and fine-grained editing results than the state-of-the-art (SOTA) methods while requiring less editing time.
arXiv Detail & Related papers (2026-03-12T11:15:11Z) - WeEdit: A Dataset, Benchmark and Glyph-Guided Framework for Text-centric Image Editing [103.68419705566146]
We present WeEdit, a systematic solution encompassing a scalable data construction pipeline, two benchmarks, and a tailored two-stage training strategy.<n>Specifically, we propose a novel HTML-based automatic editing pipeline, which generates 330K training pairs covering diverse editing operations and 15 languages.<n>On the algorithmic side, we employ glyph-guided supervised fine-tuning to inject explicit spatial and content priors, followed by a multi-objective reinforcement learning stage to align generation with instruction adherence, text clarity, and background preservation.
arXiv Detail & Related papers (2026-03-12T06:25:09Z) - InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning [60.799998743918955]
We propose a novel text-vision Interleaved Chain-of-Grounding reasoning framework for fine-grained image editing in complex real-world scenes.<n>The key insight of InterCoG is to first perform object position reasoning solely within text.<n>We also propose two auxiliary training modules: multimodal grounding reconstruction supervision and multimodal grounding reasoning alignment.
arXiv Detail & Related papers (2026-03-02T08:13:16Z) - LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents [61.91651123290512]
LangDriveCTRL is a framework for editing real-world driving videos to synthesize diverse traffic scenarios.<n>It supports both object node editing (removal, insertion and replacement) and multi-object behavior editing from a single natural-language instruction.
arXiv Detail & Related papers (2025-12-19T10:57:03Z) - ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment [13.125899660835813]
Training-free video object editing aims to achieve precise object-level manipulation, including object insertion, swapping, and deletion.<n>Existing methods suffer from two primary limitations: inaccurate inversion due to first-order solvers, and contextual conflicts caused by crude "hard" feature replacement.<n>We introduce ContextFlow, a novel training-free framework for DiT-based video object editing.
arXiv Detail & Related papers (2025-09-22T14:13:31Z) - FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
arXiv Detail & Related papers (2025-06-20T07:46:40Z) - SmartFreeEdit: Mask-Free Spatial-Aware Image Editing with Complex Instruction Understanding [46.767486063775266]
SmartFreeEdit is an end-to-end framework that integrates a multimodal large language model (MLLM) with a hypergraph-enhanced inpainting architecture.<n>Key innovations of SmartFreeEdit include region aware tokens and a mask embedding paradigm.<n>Experiments on the Reason-Edit benchmark demonstrate that SmartFreeEdit surpasses current state-of-the-art methods.
arXiv Detail & Related papers (2025-04-17T07:17:49Z) - Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing [60.730661748555214]
We introduce textbfTask-textbfOriented textbfDiffusion textbfInversion (textbfTODInv), a novel framework that inverts and edits real images tailored to specific editing tasks.
ToDInv seamlessly integrates inversion and editing through reciprocal optimization, ensuring both high fidelity and precise editability.
arXiv Detail & Related papers (2024-08-23T22:16:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.