VINCIE: Unlocking In-context Image Editing from Video

• URL: http://arxiv.org/abs/2506.10941v1
• Date: Thu, 12 Jun 2025 17:46:54 GMT
• Title: VINCIE: Unlocking In-context Image Editing from Video
• Authors: Leigang Qu, Feng Cheng, Ziyan Yang, Qi Zhao, Shanchuan Lin, Yichun Shi, Yicong Li, Wenjie Wang, Tat-Seng Chua, Lu Jiang,
• Abstract summary: In this work, we explore whether an in-context image editing model can be learned directly from videos.<n>To effectively learn from this data, we design a block-causal diffusion transformer trained on three proxy tasks.<n>Our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks.
• Score: 62.88977098700917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context image editing aims to modify images based on a contextual sequence comprising text and previously generated images. Existing methods typically depend on task-specific pipelines and expert models (e.g., segmentation and inpainting) to curate training data. In this work, we explore whether an in-context image editing model can be learned directly from videos. We introduce a scalable approach to annotate videos as interleaved multimodal sequences. To effectively learn from this data, we design a block-causal diffusion transformer trained on three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. Additionally, we propose a novel multi-turn image editing benchmark to advance research in this area. Extensive experiments demonstrate that our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks. Despite being trained exclusively on videos, our model also shows promising abilities in multi-concept composition, story generation, and chain-of-editing applications.

Related papers

• A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models [117.77807994397784]
  Image editing aims to edit the given synthetic or real image to meet the specific requirements from users. Recent significant advancement in this field is based on the development of text-to-image (T2I) diffusion models. T2I-based image editing methods significantly enhance editing performance and offer a user-friendly interface for modifying content guided by multimodal inputs.
  arXiv  Detail & Related papers  (2024-06-20T17:58:52Z)
• Unified Editing of Panorama, 3D Scenes, and Videos Through Disentangled Self-Attention Injection [60.47731445033151]
  We propose a novel unified editing framework that combines the strengths of both approaches by utilizing only a basic 2D image text-to-image (T2I) diffusion model. Experimental results confirm that our method enables editing across diverse modalities including 3D scenes, videos, and panorama images.
  arXiv  Detail & Related papers  (2024-05-27T04:44:36Z)
• Diffusion Model-Based Image Editing: A Survey [46.244266782108234]
  Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks.<n>We provide an exhaustive overview of existing methods using diffusion models for image editing.<n>To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval.
  arXiv  Detail & Related papers  (2024-02-27T14:07:09Z)
• InstructVid2Vid: Controllable Video Editing with Natural Language Instructions [97.17047888215284]
  InstructVid2Vid is an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion.
  arXiv  Detail & Related papers  (2023-05-21T03:28:13Z)
• Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts [116.05656635044357]
  We propose a generic video editing framework called Make-A-Protagonist. Specifically, we leverage multiple experts to parse source video, target visual and textual clues, and propose a visual-textual-based video generation model. Results demonstrate the versatile and remarkable editing capabilities of Make-A-Protagonist.
  arXiv  Detail & Related papers  (2023-05-15T17:59:03Z)
• Pix2Video: Video Editing using Image Diffusion [43.07444438561277]
  We investigate how to use pre-trained image models for text-guided video editing. Our method works in two simple steps: first, we use a pre-trained structure-guided (e.g., depth) image diffusion model to perform text-guided edits on an anchor frame. We demonstrate that realistic text-guided video edits are possible, without any compute-intensive preprocessing or video-specific finetuning.
  arXiv  Detail & Related papers  (2023-03-22T16:36:10Z)
• SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing [94.31103255204933]
  We propose a unified model for open-domain image editing focusing on color and tone adjustment of open-domain images. Our model learns a unified editing space that is more semantic, intuitive, and easy to manipulate. We show that by inverting image pairs into latent codes of the learned editing space, our model can be leveraged for various downstream editing tasks.
  arXiv  Detail & Related papers  (2021-11-30T23:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.