Training-and-prompt-free General Painterly Harmonization Using Image-wise Attention Sharing

• URL: http://arxiv.org/abs/2404.12900v1
• Date: Fri, 19 Apr 2024 14:13:46 GMT
• Title: Training-and-prompt-free General Painterly Harmonization Using Image-wise Attention Sharing
• Authors: Teng-Fang Hsiao, Bo-Kai Ruan, Hong-Han Shuai,
• Abstract summary: Painterly Image Harmonization aims at seamlessly blending disparate visual elements within a single coherent image. Previous approaches often encounter significant limitations due to training data constraints, the need for time-consuming fine-tuning, or reliance on additional prompts. We design a Training-and-prompt-Free General Painterly Harmonization method using image-wise attention sharing.
• Score: 20.189124622271446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Painterly Image Harmonization aims at seamlessly blending disparate visual elements within a single coherent image. However, previous approaches often encounter significant limitations due to training data constraints, the need for time-consuming fine-tuning, or reliance on additional prompts. To surmount these hurdles, we design a Training-and-prompt-Free General Painterly Harmonization method using image-wise attention sharing (TF-GPH), which integrates a novel "share-attention module". This module redefines the traditional self-attention mechanism by allowing for comprehensive image-wise attention, facilitating the use of a state-of-the-art pretrained latent diffusion model without the typical training data limitations. Additionally, we further introduce "similarity reweighting" mechanism enhances performance by effectively harnessing cross-image information, surpassing the capabilities of fine-tuning or prompt-based approaches. At last, we recognize the deficiencies in existing benchmarks and propose the "General Painterly Harmonization Benchmark", which employs range-based evaluation metrics to more accurately reflect real-world application. Extensive experiments demonstrate the superior efficacy of our method across various benchmarks. The code and web demo are available at https://github.com/BlueDyee/TF-GPH.

Related papers

• Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration [57.02757226679549]
  We introduce a training-free framework that reformulates style-guided synthesis as an in-context learning task.<n>We propose a Dynamic Semantic-Style Integration (DSSI) mechanism that reweights attention between semantic and style visual tokens.<n>Experiments show that our approach achieves high-fidelity stylization with superior semantic-style balance and visual quality.
  arXiv  Detail & Related papers  (2026-01-10T16:01:14Z)
• CompleteMe: Reference-based Human Image Completion [52.93963237043788]
  We propose CompleteMe, a novel reference-based human image completion framework. CompleteMe employs a dual U-Net architecture combined with a Region-focused Attention (RFA) Block. Our proposed method achieves superior visual quality and semantic consistency compared to existing techniques.
  arXiv  Detail & Related papers  (2025-04-28T17:59:56Z)
• CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models [21.798183378799667]
  We propose CorrFill, a training-free module designed to enhance the awareness of geometric correlations between the reference and target images. Experimental results demonstrate that CorrFill significantly enhances the performance of multiple baseline diffusion-based methods.
  arXiv  Detail & Related papers  (2025-01-04T18:31:01Z)
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling [61.14140480095604]
  This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control.
  arXiv  Detail & Related papers  (2024-08-10T08:53:41Z)
• Robust Classification by Coupling Data Mollification with Label Smoothing [25.66357344079206]
  We propose a novel approach of coupling data mollification, in the form of image noising and blurring, with label smoothing to align predicted label confidences with image degradation. We demonstrate improved robustness and uncertainty on the corrupted image benchmarks of the CIFAR and TinyImageNet datasets.
  arXiv  Detail & Related papers  (2024-06-03T16:21:29Z)
• TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability [8.896239176376488]
  This work addresses the challenge of achieving zero-shot adversarial robustness while preserving zero-shot generalization in large-scale foundation models. We propose a novel Text-Image Mutual Awareness (TIMA) method that strikes a balance between zero-shot adversarial robustness and generalization.
  arXiv  Detail & Related papers  (2024-05-27T22:10:17Z)
• DiffHarmony: Latent Diffusion Model Meets Image Harmonization [11.500358677234939]
  Diffusion models have promoted the rapid development of image-to-image translation tasks. Fine-tuning pre-trained latent diffusion models from scratch is computationally intensive. In this paper, we adapt a pre-trained latent diffusion model to the image harmonization task to generate harmonious but potentially blurry initial images.
  arXiv  Detail & Related papers  (2024-04-09T09:05:23Z)
• Attention Calibration for Disentangled Text-to-Image Personalization [12.339742346826403]
  We propose an attention calibration mechanism to improve the concept-level understanding of the T2I model. We demonstrate that our method outperforms the current state of the art in both qualitative and quantitative evaluations.
  arXiv  Detail & Related papers  (2024-03-27T13:31:39Z)
• Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [65.7968515029306]
  We propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS) A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt.
  arXiv  Detail & Related papers  (2024-02-28T06:07:07Z)
• HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models [56.112302700630806]
  We introduce an innovative algorithm named HiFi Tuner to enhance the appearance preservation of objects during personalized image generation. Key enhancements include the utilization of mask guidance, a novel parameter regularization technique, and the incorporation of step-wise subject representations. We extend our method to a novel image editing task: substituting the subject in an image through textual manipulations.
  arXiv  Detail & Related papers  (2023-11-30T02:33:29Z)
• FreePIH: Training-Free Painterly Image Harmonization with Diffusion Model [19.170302996189335]
  Our FreePIH method tames the denoising process as a plug-in module for foreground image style transfer. We make use of multi-scale features to enforce the consistency of the content and stability of the foreground objects in the latent space. Our method can surpass representative baselines by large margins.
  arXiv  Detail & Related papers  (2023-11-25T04:23:49Z)
• Zero-Shot Image Harmonization with Generative Model Prior [22.984119094424056]
  We propose a zero-shot approach to image harmonization, aiming to overcome the reliance on large amounts of synthetic composite images. We introduce a fully modularized framework inspired by human behavior. We present compelling visual results across diverse scenes and objects, along with a user study validating our approach.
  arXiv  Detail & Related papers  (2023-07-17T00:56:21Z)
• Learning from Multi-Perception Features for Real-Word Image Super-resolution [87.71135803794519]
  We propose a novel SR method called MPF-Net that leverages multiple perceptual features of input images. Our method incorporates a Multi-Perception Feature Extraction (MPFE) module to extract diverse perceptual information. We also introduce a contrastive regularization term (CR) that improves the model's learning capability.
  arXiv  Detail & Related papers  (2023-05-26T07:35:49Z)
• Exploiting Diffusion Prior for Real-World Image Super-Resolution [75.5898357277047]
  We present a novel approach to leverage prior knowledge encapsulated in pre-trained text-to-image diffusion models for blind super-resolution. By employing our time-aware encoder, we can achieve promising restoration results without altering the pre-trained synthesis model.
  arXiv  Detail & Related papers  (2023-05-11T17:55:25Z)
• Image Harmonization with Region-wise Contrastive Learning [51.309905690367835]
  We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme. Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
  arXiv  Detail & Related papers  (2022-05-27T15:46:55Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• Deep Reparametrization of Multi-Frame Super-Resolution and Denoising [167.42453826365434]
  We propose a deep reparametrization of the maximum a posteriori formulation commonly employed in multi-frame image restoration tasks. Our approach is derived by introducing a learned error metric and a latent representation of the target image. We validate our approach through comprehensive experiments on burst denoising and burst super-resolution datasets.
  arXiv  Detail & Related papers  (2021-08-18T17:57:02Z)
• SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
  We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
  arXiv  Detail & Related papers  (2021-08-15T19:51:33Z)
• Towards Unsupervised Sketch-based Image Retrieval [126.77787336692802]
  We introduce a novel framework that simultaneously performs unsupervised representation learning and sketch-photo domain alignment. Our framework achieves excellent performance in the new unsupervised setting, and performs comparably or better than state-of-the-art in the zero-shot setting.
  arXiv  Detail & Related papers  (2021-05-18T02:38:22Z)
• On Feature Normalization and Data Augmentation [55.115583969831]
  Moment Exchange encourages the model to utilize the moment information also for recognition models. We replace the moments of the learned features of one training image by those of another, and also interpolate the target labels. As our approach is fast, operates entirely in feature space, and mixes different signals than prior methods, one can effectively combine it with existing augmentation approaches.
  arXiv  Detail & Related papers  (2020-02-25T18:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.