InstructVEdit: A Holistic Approach for Instructional Video Editing

• URL: http://arxiv.org/abs/2503.17641v1
• Date: Sat, 22 Mar 2025 04:12:20 GMT
• Title: InstructVEdit: A Holistic Approach for Instructional Video Editing
• Abstract summary: InstructVEdit is a full-cycle instructional video editing approach that establishes a reliable dataset curation workflow.<n>It incorporates two model architectural improvements to enhance edit quality while preserving temporal consistency.<n>It also proposes an iterative refinement strategy leveraging real-world data to enhance generalization and minimize train-test discrepancies.
• Score: 28.13673601495108
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video editing according to instructions is a highly challenging task due to the difficulty in collecting large-scale, high-quality edited video pair data. This scarcity not only limits the availability of training data but also hinders the systematic exploration of model architectures and training strategies. While prior work has improved specific aspects of video editing (e.g., synthesizing a video dataset using image editing techniques or decomposed video editing training), a holistic framework addressing the above challenges remains underexplored. In this study, we introduce InstructVEdit, a full-cycle instructional video editing approach that: (1) establishes a reliable dataset curation workflow to initialize training, (2) incorporates two model architectural improvements to enhance edit quality while preserving temporal consistency, and (3) proposes an iterative refinement strategy leveraging real-world data to enhance generalization and minimize train-test discrepancies. Extensive experiments show that InstructVEdit achieves state-of-the-art performance in instruction-based video editing, demonstrating robust adaptability to diverse real-world scenarios. Project page: https://o937-blip.github.io/InstructVEdit.

Related papers

• LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing [45.1196749483108]
  We propose LIVE, a joint training framework that leverages large-scale, high-quality image editing data alongside video datasets to bolster editing capabilities.<n>We introduce a frame-wise token noise strategy, which treats the latents of specific frames as reasoning tokens, leveraging large pretrained video generative models to create plausible temporal transformations.<n>We curate a comprehensive evaluation benchmark encompassing over 60 challenging tasks that are prevalent in image editing but scarce in existing video datasets.
  arXiv  Detail & Related papers  (2026-04-18T15:09:01Z)
• Versatile Editing of Video Content, Actions, and Dynamics without Training [57.167192591765456]
  We introduce DynaEdit, a training-free editing method that unlocks versatile video editing capabilities with pretrained text-to-video flow models.<n>We show that DynaEdit achieves state-of-the-art results on complex text-based video editing tasks, including modifying actions, inserting objects that interact with the scene, and introducing global effects.
  arXiv  Detail & Related papers  (2026-03-18T17:50:56Z)
• Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance [55.32799307123252]
  We introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets.<n>We propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance.
  arXiv  Detail & Related papers  (2026-03-02T18:46:28Z)
• Region-Constraint In-Context Generation for Instructional Video Editing [91.27224696009755]
  We present ReCo, a new instructional video editing paradigm that delves into constraint modeling between editing and non-editing regions during in-context generation.<n>We propose a large-scale, high-quality video editing dataset, i.e., ReCo-Data, comprising 500K instruction-video pairs to benefit model training.
  arXiv  Detail & Related papers  (2025-12-19T14:49:30Z)
• In-Context Learning with Unpaired Clips for Instruction-based Video Editing [51.943707933717185]
  We introduce a low-cost pretraining strategy for instruction-based video editing.<n>Our framework first pretrains on approximately 1M real video clips to learn basic editing concepts.<n>Our method surpasses existing instruction-based video editing approaches in both instruction alignment and visual fidelity.
  arXiv  Detail & Related papers  (2025-10-16T13:02:11Z)
• EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning [58.53074381801114]
  We introduce EditVerse, a unified framework for image and video generation and editing within a single model.<n>By representing all modalities, i.e. text, image, and video, as a unified token sequence, EditVerse leverages self-attention to achieve robust in-context learning.<n>We present EditVerseBench, the first benchmark for instruction-based video editing covering diverse tasks and resolutions.
  arXiv  Detail & Related papers  (2025-09-24T17:59:30Z)
• InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction [10.855393943204728]
  We present a high-quality Instruction-based Video Editing dataset with 1M triplets, namely InsViE-1M. We first curate high-resolution and high-quality source videos and images, then design an effective editing-filtering pipeline to construct high-quality editing triplets for model training.
  arXiv  Detail & Related papers  (2025-03-26T07:30:58Z)
• VEGGIE: Instructional Editing and Reasoning of Video Concepts with Grounded Generation [67.31149310468801]
  We introduce VEGGIE, a simple end-to-end framework that unifies video concept editing, grounding, and reasoning based on diverse user instructions.<n> VEGGIE shows strong performance in instructional video editing with different editing skills, outperforming the best instructional baseline as a versatile model.
  arXiv  Detail & Related papers  (2025-03-18T15:31:12Z)
• Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists [17.451911831989293]
  We introduce Senorita-2M, a high-quality video editing dataset.<n>It is built by crafting four high-quality, specialized video editing models.<n>We propose a filtering pipeline to eliminate poorly edited video pairs.
  arXiv  Detail & Related papers  (2025-02-10T17:58:22Z)
• DreamOmni: Unified Image Generation and Editing [51.45871494724542]
  We introduce Dream Omni, a unified model for image generation and editing. For training, Dream Omni jointly trains T2I generation and downstream tasks. This collaboration significantly boosts editing performance.
  arXiv  Detail & Related papers  (2024-12-22T17:17:28Z)
• SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing [50.098005973600024]
  We propose a novel video generation and editing system powered by our Semantic Planning Agent (SPAgent)<n>SPAgent bridges the gap between diverse user intents and the effective utilization of existing generative models.<n> Experimental results demonstrate that the SPAgent effectively coordinates models to generate or edit videos.
  arXiv  Detail & Related papers  (2024-11-28T08:07:32Z)
• A Reinforcement Learning-Based Automatic Video Editing Method Using Pre-trained Vision-Language Model [10.736207095604414]
  We propose a two-stage scheme for general editing. Firstly, unlike previous works that extract scene-specific features, we leverage the pre-trained Vision-Language Model (VLM) We also propose a Reinforcement Learning (RL)-based editing framework to formulate the editing problem and train the virtual editor to make better sequential editing decisions.
  arXiv  Detail & Related papers  (2024-11-07T18:20:28Z)
• Zero-Shot Video Editing through Adaptive Sliding Score Distillation [51.57440923362033]
  This study proposes a novel paradigm of video-based score distillation, facilitating direct manipulation of original video content. We propose an Adaptive Sliding Score Distillation strategy, which incorporates both global and local video guidance to reduce the impact of editing errors.
  arXiv  Detail & Related papers  (2024-06-07T12:33:59Z)
• ReVideo: Remake a Video with Motion and Content Control [67.5923127902463]
  We present a novel attempt to Remake a Video (VideoRe) which allows precise video editing in specific areas through the specification of both content and motion. VideoRe addresses a new task involving the coupling and training imbalance between content and motion control. Our method can also seamlessly extend these applications to multi-area editing without modifying specific training, demonstrating its flexibility and robustness.
  arXiv  Detail & Related papers  (2024-05-22T17:46:08Z)
• InstructBrush: Learning Attention-based Instruction Optimization for Image Editing [54.07526261513434]
  InstructBrush is an inversion method for instruction-based image editing methods. It extracts editing effects from image pairs as editing instructions, which are further applied for image editing. Our approach achieves superior performance in editing and is more semantically consistent with the target editing effects.
  arXiv  Detail & Related papers  (2024-03-27T15:03:38Z)
• EffiVED:Efficient Video Editing via Text-instruction Diffusion Models [9.287394166165424]
  We introduce EffiVED, an efficient diffusion-based model that supports instruction-guided video editing. We transform vast image editing datasets and open-world videos into a high-quality dataset for training EffiVED.
  arXiv  Detail & Related papers  (2024-03-18T08:42:08Z)
• InstructVid2Vid: Controllable Video Editing with Natural Language Instructions [97.17047888215284]
  InstructVid2Vid is an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion.
  arXiv  Detail & Related papers  (2023-05-21T03:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.