Tuning-Free Image Editing with Fidelity and Editability via Unified Latent Diffusion Model

• URL: http://arxiv.org/abs/2504.05594v1
• Date: Tue, 08 Apr 2025 01:02:50 GMT
• Title: Tuning-Free Image Editing with Fidelity and Editability via Unified Latent Diffusion Model
• Abstract summary: We introduce UnifyEdit, a tuning-free method that performs diffusion latent optimization.<n>We develop two attention-based constraints: a self-attention (SA) preservation constraint for structural fidelity, and a cross-attention (CA) alignment constraint to enhance text alignment.<n>Our approach achieves a robust balance between structure preservation and text alignment across various editing tasks, outperforming other state-of-the-art methods.
• Score: 60.82962950960996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Balancing fidelity and editability is essential in text-based image editing (TIE), where failures commonly lead to over- or under-editing issues. Existing methods typically rely on attention injections for structure preservation and leverage the inherent text alignment capabilities of pre-trained text-to-image (T2I) models for editability, but they lack explicit and unified mechanisms to properly balance these two objectives. In this work, we introduce UnifyEdit, a tuning-free method that performs diffusion latent optimization to enable a balanced integration of fidelity and editability within a unified framework. Unlike direct attention injections, we develop two attention-based constraints: a self-attention (SA) preservation constraint for structural fidelity, and a cross-attention (CA) alignment constraint to enhance text alignment for improved editability. However, simultaneously applying both constraints can lead to gradient conflicts, where the dominance of one constraint results in over- or under-editing. To address this challenge, we introduce an adaptive time-step scheduler that dynamically adjusts the influence of these constraints, guiding the diffusion latent toward an optimal balance. Extensive quantitative and qualitative experiments validate the effectiveness of our approach, demonstrating its superiority in achieving a robust balance between structure preservation and text alignment across various editing tasks, outperforming other state-of-the-art methods. The source code will be available at https://github.com/CUC-MIPG/UnifyEdit.

Related papers

• Conditioning Matters: Stabilizing Inversion and Attention in Diffusion Image Editing [27.02750589908665]
  Inversion-based image editing offers flexible and training-free control but still struggles with inversion accuracy and the trade-off between editing fidelity and background preservation.<n>We show that the precision of textual conditioning influences inversion stability by modulating the geometry of the diffusion velocity field.<n>We propose SimEdit, a conditioning-aware framework with two complementary components: conditioning refinement and cross-branch attention control.
  arXiv  Detail & Related papers  (2026-06-12T05:13:01Z)
• From Competition to Coopetition: Coopetitive Training-Free Image Editing Based on Text Guidance [11.574335632043491]
  CoEdit is a novel zero-shot framework that transforms attention control from competition to coopetitive negotiation.<n>We present Dual-Entropy Attention Manipulation, which quantifies directional entropic interactions between branches to reformulate attention control as a harmony-maximization problem.<n>We also present Entropic Latent Refinement mechanism to dynamically adjust latent representations over time, minimizing accumulated editing errors.
  arXiv  Detail & Related papers  (2026-04-17T11:10:22Z)
• The Devil is in Attention Sharing: Improving Complex Non-rigid Image Editing Faithfulness via Attention Synergy [71.39358554558667]
  We introduce SynPS, a method that Synergistically leverages Positional embeddings and Semantic information for faithful non-rigid image editing.<n>We propose an editing measurement that quantifies the required editing magnitude at each denoising step.<n>By adaptively integrating positional and semantic cues, SynPS effectively avoids both over- and under-editing.
  arXiv  Detail & Related papers  (2025-12-16T14:08:00Z)
• ConsistEdit: Highly Consistent and Precise Training-free Visual Editing [17.162316662697965]
  We propose ConsistEdit, a novel attention control method specifically tailored for MM-DiT.<n>It incorporates vision-only attention control, mask-guided pre-attention fusion, and differentiated manipulation of the query, key, and value tokens.<n>It achieves state-of-the-art performance across a wide range of image and video editing tasks, including both structure-consistent and structure-inconsistent scenarios.
  arXiv  Detail & Related papers  (2025-10-20T17:59:52Z)
• Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing [76.44219733285898]
  Kontinuous Kontext is an instruction-driven editing model that provides a new dimension of control over edit strength.<n>A lightweight projector network maps the input scalar and the edit instruction to coefficients in the model's modulation space.<n>For training our model, we synthesize a diverse dataset of image-edit-instruction-strength quadruplets using existing generative models.
  arXiv  Detail & Related papers  (2025-10-09T17:51:03Z)
• LatentEdit: Adaptive Latent Control for Consistent Semantic Editing [24.414252461549555]
  LatentEdit is an adaptive latent fusion framework that combines the current latent code with a reference latent code inverted from the source image.<n>Our proposed LatentEdit achieves an optimal balance between fidelity and editability, outperforming the state-of-the-art method even in 8-15 steps.
  arXiv  Detail & Related papers  (2025-08-30T15:47:03Z)
• LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editing [0.276240219662896]
  We introduce LORE, a training-free and efficient image editing method.<n>LORE directly optimize the inverted noise, addressing the core limitations in generalization and controllability of existing approaches.<n> Experimental results show that LORE significantly outperforms strong baselines in terms of semantic alignment, image quality, and background fidelity.
  arXiv  Detail & Related papers  (2025-08-05T06:45:04Z)
• Stable Score Distillation [45.48460025487433]
  We introduce Stable Score Distillation (SSD), a streamlined framework that enhances stability and alignment in the editing process.<n>Our method achieves state-of-the-art results in 2D and 3D editing tasks, including NeRF and text-driven style edits, with faster convergence and reduced complexity.
  arXiv  Detail & Related papers  (2025-07-12T07:14:00Z)
• Improving Diffusion-Based Image Editing Faithfulness via Guidance and Scheduling [1.8876415010297893]
  In image editing, two crucial aspects are editability, which determines the extent of modification, and faithfulness, which reflects how well unaltered elements are preserved.<n>We propose Faithfulness Guidance and Scheduling (FGS), which enhances faithfulness with minimal impact on editability.<n> Experimental results demonstrate that FGS achieves superior faithfulness while maintaining editability.
  arXiv  Detail & Related papers  (2025-06-26T06:46:03Z)
• Control and Realism: Best of Both Worlds in Layout-to-Image without Training [59.16447569868382]
  We present WinWinLay, a training-free method for layout-to-Image generation.<n>We propose two key strategies, Non-local Attention Energy and Adaptive Update, that collaboratively enhance control precision and realism.<n>WinWinLay excels in controlling element placement and achieving photorealistic visual fidelity, outperforming the current state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-06-18T15:39:02Z)
• DisProtEdit: Exploring Disentangled Representations for Multi-Attribute Protein Editing [48.819599672346136]
  DisProtEdit is a controllable protein editing framework that leverages dual-channel natural language supervision.<n>DisProtEdit explicitly separates semantic factors, enabling modular and interpretable control.
  arXiv  Detail & Related papers  (2025-06-17T06:12:18Z)
• Training-Free Text-Guided Image Editing with Visual Autoregressive Model [46.201510044410995]
  We propose a novel text-guided image editing framework based on Visual AutoRegressive modeling.<n>Our method eliminates the need for explicit inversion while ensuring precise and controlled modifications.<n>Our framework operates in a training-free manner and achieves high-fidelity editing with faster inference speeds.
  arXiv  Detail & Related papers  (2025-03-31T09:46:56Z)
• DCEdit: Dual-Level Controlled Image Editing via Precisely Localized Semantics [71.78350994830885]
  We present a novel approach to improving text-guided image editing using diffusion-based models.<n>Our method uses visual and textual self-attention to enhance the cross-attention map, which can serve as a regional cues to improve editing performance.<n>To fully compare our methods with other DiT-based approaches, we construct the RW-800 benchmark, featuring high resolution images, long descriptive texts, real-world images, and a new text editing task.
  arXiv  Detail & Related papers  (2025-03-21T02:14:03Z)
• UIP2P: Unsupervised Instruction-based Image Editing via Cycle Edit Consistency [69.33072075580483]
  We propose an unsupervised model for instruction-based image editing that eliminates the need for ground-truth edited images during training.<n>Our method addresses these challenges by introducing a novel editing mechanism called Cycle Edit Consistency ( CEC)<n> CEC applies forward and backward edits in one training step and enforces consistency in image and attention spaces.
  arXiv  Detail & Related papers  (2024-12-19T18:59:58Z)
• Uniform Attention Maps: Boosting Image Fidelity in Reconstruction and Editing [66.48853049746123]
  We analyze reconstruction from a structural perspective and propose a novel approach that replaces traditional cross-attention with uniform attention maps.<n>Our method effectively minimizes distortions caused by varying text conditions during noise prediction.<n> Experimental results demonstrate that our approach not only excels in achieving high-fidelity image reconstruction but also performs robustly in real image composition and editing scenarios.
  arXiv  Detail & Related papers  (2024-11-29T12:11:28Z)
• Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theory [33.78620829249978]
  Text-to-image (T2I) diffusion models have revolutionized generative modeling by producing high-fidelity, diverse, and visually realistic images. Recent attention-based methods have improved object inclusion and linguistic binding, but still face challenges such as attribute misbinding. We propose a Bayesian approach that designs custom priors over attention distributions to enforce desirable properties. Our approach treats the attention mechanism as an interpretable component, enabling fine-grained control and improved attribute-object alignment.
  arXiv  Detail & Related papers  (2024-11-25T10:57:48Z)
• Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing [60.730661748555214]
  We introduce textbfTask-textbfOriented textbfDiffusion textbfInversion (textbfTODInv), a novel framework that inverts and edits real images tailored to specific editing tasks. ToDInv seamlessly integrates inversion and editing through reciprocal optimization, ensuring both high fidelity and precise editability.
  arXiv  Detail & Related papers  (2024-08-23T22:16:34Z)
• E4C: Enhance Editability for Text-Based Image Editing by Harnessing Efficient CLIP Guidance [13.535394339438428]
  Diffusion-based image editing is a composite process of preserving the source image content and generating new content or applying modifications. We propose a zero-shot image editing method, named textbfEnhance textbfEditability for text-based image textbfEditing via textbfCLIP guidance.
  arXiv  Detail & Related papers  (2024-03-15T09:26:48Z)
• Doubly Abductive Counterfactual Inference for Text-based Image Editing [130.46583155383735]
  We study text-based image editing (TBIE) of a single image by counterfactual inference. We propose a Doubly Abductive Counterfactual inference framework (DAC) Our DAC achieves a good trade-off between editability and fidelity.
  arXiv  Detail & Related papers  (2024-03-05T13:59:21Z)
• BARET : Balanced Attention based Real image Editing driven by Target-text Inversion [36.59406959595952]
  We propose a novel editing technique that only requires an input image and target text for various editing types including non-rigid edits without fine-tuning diffusion model. Our method contains three novelties: (I) Targettext Inversion Schedule (TTIS) is designed to fine-tune the input target text embedding to achieve fast image reconstruction without image caption and acceleration of convergence; (II) Progressive Transition Scheme applies progressive linear approaches between target text embedding and its fine-tuned version to generate transition embedding for maintaining non-rigid editing capability; (III) Balanced Attention Module (BAM) balances the tradeoff between textual description and image semantics
  arXiv  Detail & Related papers  (2023-12-09T07:18:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.