VicEdit: Learning to Edit Videos from Visual In-Context Examples
Abstract Overview
This paper introduces Visual In-context Editing, a video editing paradigm that supplements text instructions with visual examples in the form of a single image, an image pair, or a video pair. To support this setting, the authors curate VicEdit-400K, a 400K-sample dataset spanning three reference modalities and ten task types, built through automated instruction synthesis, reference generation, and multi-stage quality filtering. They also propose VicEdit, a unified framework that combines Modality-Adaptive Semantic Distillation (MASD) for extracting modality-specific visual semantics and Dual-Context Injection (DCI) for aligning visual context with textual instructions inside a Video DiT. Experiments on OpenVE-Ext and the new VicEdit-Bench evaluate both standard instruction-based editing and visual in-context editing. The reported results indicate that adding visual exemplars improves controllability and performance over text-only editing approaches.
Novelty
The main novelty is the formulation of visual in-context video editing as a unified multimodal editing problem that supports heterogeneous references across single-image, image-pair, and video-pair inputs. The work also contributes the first large-scale dataset for this setting, VicEdit-400K, together with a unified architecture that explicitly adapts semantic extraction and instruction conditioning to the reference modality.
Results
On the base editing benchmark OpenVE-Ext, VicEdit achieves the best reported overall score of 3.28, outperforming the listed baselines. On VicEdit-Bench for visual in-context editing, it reaches an overall score of 3.45, exceeding the second-best method, Kiwi-Edit, at 2.97. Ablation studies further show that both MASD and DCI contribute to performance, improving the overall VicEdit-Bench score from 3.02 in the base variant to 3.45 when both modules are used.
Key Points
- VicEdit extends video editing from text-only instructions to multimodal visual guidance using single-image, image-pair, and video-pair references.
- The paper introduces VicEdit-400K, a 400K-sample dataset and VicEdit-Bench, a benchmark designed to train and evaluate visual in-context video editing.
- The proposed MASD and DCI modules improve reference-aware semantic extraction and text-visual alignment, supporting stronger results in both conventional and visual in-context editing tasks.