Rethinking Where to Edit: Task-Aware Localization for Instruction-Based Image Editing

• URL: http://arxiv.org/abs/2604.20258v1
• Date: Wed, 22 Apr 2026 07:08:01 GMT
• Title: Rethinking Where to Edit: Task-Aware Localization for Instruction-Based Image Editing
• Abstract summary: We propose a training-free, task-aware edit localization framework for instruction-based image editing.<n>Our framework consistently improves non-edit region consistency while maintaining strong instruction-following performance.
• Score: 30.23368654238746
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Instruction-based image editing (IIE) aims to modify images according to textual instructions while preserving irrelevant content. Despite recent advances in diffusion transformers, existing methods often suffer from over-editing, introducing unintended changes to regions unrelated to the desired edit. We identify that this limitation arises from the lack of an explicit mechanism for edit localization. In particular, different editing operations (e.g., addition, removal and replacement) induce distinct spatial patterns, yet current IIE models typically treat localization in a task-agnostic manner. To address this limitation, we propose a training-free, task-aware edit localization framework that exploits the intrinsic source and target image streams within IIE models. For each image stream, We first obtain attention-based edit cues, and then construct feature centroids based on these attentive cues to partition tokens into edit and non-edit regions. Based on the observation that optimal localization is inherently task-dependent, we further introduce a unified mask construction strategy that selectively leverages source and target image streams for different editing tasks. We provide a systematic analysis for our proposed insights and approaches. Extensive experiments on EdiVal-Bench demonstrate our framework consistently improves non-edit region consistency while maintaining strong instruction-following performance on top of powerful recent image editing backbones, including Step1X-Edit and Qwen-Image-Edit.

Related papers

• WhereEdit: Mask-aware Local Latent Editing for One-Step Image Editing [16.758428220880734]
  One-step text-to-image (T2I) models enable efficient image synthesis and provide new opportunities for real-time image editing.<n>Existing one-step editing methods rely on text conditioning for semantic transformation, lacking explicit spatial control over textitwhere to edit.<n>We propose textbfWhereEdit, a framework that reformulates one-step editing as localized adaptive editing.
  arXiv  Detail & Related papers  (2026-07-23T03:11:25Z)
• Edit-GRPO: A Locality-Preserving Policy Optimization Framework for Image Editing [95.29911956874389]
  We propose Edit-GRPO, preserving Locality while optimizing image editing.<n>By assigning region-specific optimization signals to edit and non-edit areas, Edit-GRPO aligns policy updates with the spatial structure of editing tasks.<n>This design effectively suppresses common artifacts such as context distortion and boundary inconsistency.
  arXiv  Detail & Related papers  (2026-05-16T12:05:39Z)
• LocateEdit-Bench: A Benchmark for Instruction-Based Editing Localization [21.62979058692505]
  We propose a large-scale dataset comprising $231$K edited images to benchmark forgery localization methods.<n>Our dataset incorporates four cutting-edge editing models and covers three common edit types.<n>Our work establishes a foundation to keep pace with the evolving landscape of image editing, thereby facilitating the development of effective methods for future forgery localization.
  arXiv  Detail & Related papers  (2026-02-05T12:01:09Z)
• CPAM: Context-Preserving Adaptive Manipulation for Zero-Shot Real Image Editing [24.68304617869157]
  Context-Preserving Adaptive Manipulation (CPAM) is a novel framework for complicated, non-rigid real image editing.<n>We develop a preservation adaptation module that adjusts self-attention mechanisms to preserve and independently control the object and background effectively.<n>We also introduce various mask-guidance strategies to facilitate diverse image manipulation tasks in a simple manner.
  arXiv  Detail & Related papers  (2025-06-23T09:19:38Z)
• Image Editing As Programs with Diffusion Models [69.05164729625052]
  We introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture.<n>IEAP approaches instructional editing through a reductionist lens, decomposing complex editing instructions into sequences of atomic operations.<n>Our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions.
  arXiv  Detail & Related papers  (2025-06-04T16:57:24Z)
• X-Edit: Detecting and Localizing Edits in Images Altered by Text-Guided Diffusion Models [3.610796534465868]
  Experimental results demonstrate that X-Edit accurately localizes edits in images altered by text-guided diffusion models.<n>This highlights X-Edit's potential as a robust forensic tool for detecting and pinpointing manipulations introduced by advanced image editing techniques.
  arXiv  Detail & Related papers  (2025-05-16T23:29:38Z)
• LOCATEdit: Graph Laplacian Optimized Cross Attention for Localized Text-Guided Image Editing [6.057289837472806]
  Text-guided image editing aims to modify specific regions of an image according to natural language instructions.<n>Since cross-attention mechanisms focus on semantic relevance, they struggle to maintain the image integrity.<n>We introduce LOCATEdit, which enhances cross-attention maps through a graph-based approach.
  arXiv  Detail & Related papers  (2025-03-27T14:32:17Z)
• PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models [80.98455219375862]
  We present the first text-based image editing approach for object parts based on pre-trained diffusion models.<n>Our approach is preferred by users 66-90% of the time in conducted user studies.
  arXiv  Detail & Related papers  (2025-02-06T13:08:43Z)
• EditScout: Locating Forged Regions from Diffusion-based Edited Images with Multimodal LLM [50.054404519821745]
  We present a novel framework that integrates a multimodal Large Language Model for enhanced reasoning capabilities.<n>Our framework achieves promising results on MagicBrush, AutoSplice, and PerfBrush datasets.<n> Notably, our method excels on the PerfBrush dataset, a self-constructed test set featuring previously unseen types of edits.
  arXiv  Detail & Related papers  (2024-12-05T02:05:33Z)
• LIME: Localized Image Editing via Attention Regularization in Diffusion Models [69.33072075580483]
  This paper introduces LIME for localized image editing in diffusion models.<n>LIME does not require user-specified regions of interest (RoI) or additional text input, but rather employs features from pre-trained methods and a straightforward clustering method to obtain precise editing mask.<n>We propose a novel cross-attention regularization technique that penalizes unrelated cross-attention scores in the RoI during the denoising steps, ensuring localized edits.
  arXiv  Detail & Related papers  (2023-12-14T18:59:59Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.