Pathways on the Image Manifold: Image Editing via Video Generation

• URL: http://arxiv.org/abs/2411.16819v1
• Date: Mon, 25 Nov 2024 16:41:45 GMT
• Title: Pathways on the Image Manifold: Image Editing via Video Generation
• Authors: Noam Rotstein, Gal Yona, Daniel Silver, Roy Velich, David Bensaïd, Ron Kimmel,
• Abstract summary: We reformulate image editing as a temporal process, using pretrained video models to create smooth transitions from the original image to the desired edit. Our approach achieves state-of-the-art results on text-based image editing, demonstrating significant improvements in both edit accuracy and image preservation.
• Score: 11.891831122571995
• License:
• Abstract: Recent advances in image editing, driven by image diffusion models, have shown remarkable progress. However, significant challenges remain, as these models often struggle to follow complex edit instructions accurately and frequently compromise fidelity by altering key elements of the original image. Simultaneously, video generation has made remarkable strides, with models that effectively function as consistent and continuous world simulators. In this paper, we propose merging these two fields by utilizing image-to-video models for image editing. We reformulate image editing as a temporal process, using pretrained video models to create smooth transitions from the original image to the desired edit. This approach traverses the image manifold continuously, ensuring consistent edits while preserving the original image's key aspects. Our approach achieves state-of-the-art results on text-based image editing, demonstrating significant improvements in both edit accuracy and image preservation.

Related papers

• SeedEdit: Align Image Re-Generation to Image Editing [23.461716630027436]
  We introduce SeedEdit, a diffusion model that is able to revise a given image with any text prompt. We start from a weak generator (text-to-image model) that creates diverse pairs between such two directions. SeedEdit can achieve more diverse and stable editing capability over prior image editing methods.
  arXiv  Detail & Related papers  (2024-11-11T03:06:26Z)
• ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models [11.830273909934688]
  Modern Text-to-Image (T2I) Diffusion models have revolutionized image editing by enabling the generation of high-quality images. We propose ReEdit, a modular and efficient end-to-end framework that captures edits in both text and image modalities. Our results demonstrate that ReEdit consistently outperforms contemporary approaches both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2024-11-06T15:19:24Z)
• Unified Editing of Panorama, 3D Scenes, and Videos Through Disentangled Self-Attention Injection [60.47731445033151]
  We propose a novel unified editing framework that combines the strengths of both approaches by utilizing only a basic 2D image text-to-image (T2I) diffusion model. Experimental results confirm that our method enables editing across diverse modalities including 3D scenes, videos, and panorama images.
  arXiv  Detail & Related papers  (2024-05-27T04:44:36Z)
• I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models [18.36472998650704]
  We introduce a novel and generic solution that extends the applicability of image editing tools to videos by propagating edits from a single frame to the entire video using a pre-trained image-to-video model. Our method, dubbed I2VEdit, adaptively preserves the visual and motion integrity of the source video depending on the extent of the edits.
  arXiv  Detail & Related papers  (2024-05-26T11:47:40Z)
• DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing [66.43179841884098]
  Large-scale Text-to-Image (T2I) diffusion models have revolutionized image generation over the last few years. We propose DiffEditor to rectify two weaknesses in existing diffusion-based image editing. Our method can efficiently achieve state-of-the-art performance on various fine-grained image editing tasks.
  arXiv  Detail & Related papers  (2024-02-04T18:50:29Z)
• LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance [0.0]
  LEDITS is a combined lightweight approach for real-image editing, incorporating the Edit Friendly DDPM inversion technique with Semantic Guidance. This approach achieves versatile edits, both subtle and extensive as well as alterations in composition and style, while requiring no optimization nor extensions to the architecture.
  arXiv  Detail & Related papers  (2023-07-02T09:11:09Z)
• ReGeneration Learning of Diffusion Models with Rich Prompts for Zero-Shot Image Translation [8.803251014279502]
  Large-scale text-to-image models have demonstrated amazing ability to synthesize diverse and high-fidelity images. Current models can impose significant changes to the original image content during the editing process. We propose ReGeneration learning in an image-to-image Diffusion model (ReDiffuser)
  arXiv  Detail & Related papers  (2023-05-08T12:08:12Z)
• MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing [54.712205852602736]
  We develop MasaCtrl, a tuning-free method to achieve consistent image generation and complex non-rigid image editing simultaneously. Specifically, MasaCtrl converts existing self-attention in diffusion models into mutual self-attention, so that it can query correlated local contents and textures from source images for consistency. Extensive experiments show that the proposed MasaCtrl can produce impressive results in both consistent image generation and complex non-rigid real image editing.
  arXiv  Detail & Related papers  (2023-04-17T17:42:19Z)
• StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing [86.92711729969488]
  We exploit the amazing capacities of pretrained diffusion models for the editing of images. They either finetune the model, or invert the image in the latent space of the pretrained model. They suffer from two problems: Unsatisfying results for selected regions, and unexpected changes in nonselected regions.
  arXiv  Detail & Related papers  (2023-03-28T00:16:45Z)
• Zero-shot Image-to-Image Translation [57.46189236379433]
  We propose pix2pix-zero, an image-to-image translation method that can preserve the original image without manual prompting. We propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. Our method does not need additional training for these edits and can directly use the existing text-to-image diffusion model.
  arXiv  Detail & Related papers  (2023-02-06T18:59:51Z)
• Task-agnostic Temporally Consistent Facial Video Editing [84.62351915301795]
  We propose a task-agnostic, temporally consistent facial video editing framework. Based on a 3D reconstruction model, our framework is designed to handle several editing tasks in a more unified and disentangled manner. Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.
  arXiv  Detail & Related papers  (2020-07-03T02:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.