EditWorld: Simulating World Dynamics for Instruction-Following Image Editing

• URL: http://arxiv.org/abs/2405.14785v1
• Date: Thu, 23 May 2024 16:54:17 GMT
• Title: EditWorld: Simulating World Dynamics for Instruction-Following Image Editing
• Authors: Ling Yang, Bohan Zeng, Jiaming Liu, Hong Li, Minghao Xu, Wentao Zhang, Shuicheng Yan,
• Abstract summary: Diffusion models have significantly improved the performance of image editing. We introduce world-instructed image editing, which defines and categorizes the instructions grounded by various world scenarios. Our method significantly outperforms existing editing methods in this new task.
• Score: 68.6224340373457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have significantly improved the performance of image editing. Existing methods realize various approaches to achieve high-quality image editing, including but not limited to text control, dragging operation, and mask-and-inpainting. Among these, instruction-based editing stands out for its convenience and effectiveness in following human instructions across diverse scenarios. However, it still focuses on simple editing operations like adding, replacing, or deleting, and falls short of understanding aspects of world dynamics that convey the realistic dynamic nature in the physical world. Therefore, this work, EditWorld, introduces a new editing task, namely world-instructed image editing, which defines and categorizes the instructions grounded by various world scenarios. We curate a new image editing dataset with world instructions using a set of large pretrained models (e.g., GPT-3.5, Video-LLava and SDXL). To enable sufficient simulation of world dynamics for image editing, our EditWorld trains model in the curated dataset, and improves instruction-following ability with designed post-edit strategy. Extensive experiments demonstrate our method significantly outperforms existing editing methods in this new task. Our dataset and code will be available at https://github.com/YangLing0818/EditWorld

Related papers

• InstructBrush: Learning Attention-based Instruction Optimization for Image Editing [54.07526261513434]
  InstructBrush is an inversion method for instruction-based image editing methods. It extracts editing effects from image pairs as editing instructions, which are further applied for image editing. Our approach achieves superior performance in editing and is more semantically consistent with the target editing effects.
  arXiv  Detail & Related papers  (2024-03-27T15:03:38Z)
• Emu Edit: Precise Image Editing via Recognition and Generation Tasks [62.95717180730946]
  We present Emu Edit, a multi-task image editing model which sets state-of-the-art results in instruction-based image editing. We train it to multi-task across an unprecedented range of tasks, such as region-based editing, free-form editing, and Computer Vision tasks. We show that Emu Edit can generalize to new tasks, such as image inpainting, super-resolution, and compositions of editing tasks, with just a few labeled examples.
  arXiv  Detail & Related papers  (2023-11-16T18:55:58Z)
• Guiding Instruction-based Image Editing via Multimodal Large Language Models [102.82211398699644]
  Multimodal large language models (MLLMs) show promising capabilities in cross-modal understanding and visual-aware response generation. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE) MGIE learns to derive expressive instructions and provides explicit guidance.
  arXiv  Detail & Related papers  (2023-09-29T10:01:50Z)
• LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance [0.0]
  LEDITS is a combined lightweight approach for real-image editing, incorporating the Edit Friendly DDPM inversion technique with Semantic Guidance. This approach achieves versatile edits, both subtle and extensive as well as alterations in composition and style, while requiring no optimization nor extensions to the architecture.
  arXiv  Detail & Related papers  (2023-07-02T09:11:09Z)
• Learning by Planning: Language-Guided Global Image Editing [53.72807421111136]
  We develop a text-to-operation model to map the vague editing language request into a series of editing operations. The only supervision in the task is the target image, which is insufficient for a stable training of sequential decisions. We propose a novel operation planning algorithm to generate possible editing sequences from the target image as pseudo ground truth.
  arXiv  Detail & Related papers  (2021-06-24T16:30:03Z)
• A Benchmark and Baseline for Language-Driven Image Editing [81.74863590492663]
  We first present a new language-driven image editing dataset that supports both local and global editing. Our new method treats each editing operation as a sub-module and can automatically predict operation parameters. We believe our work, including both the benchmark and the baseline, will advance the image editing area towards a more general and free-form level.
  arXiv  Detail & Related papers  (2020-10-05T20:51:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.