Learning Global-aware Kernel for Image Harmonization
- URL: http://arxiv.org/abs/2305.11676v2
- Date: Thu, 17 Aug 2023 12:16:14 GMT
- Title: Learning Global-aware Kernel for Image Harmonization
- Authors: Xintian Shen, Jiangning Zhang, Jun Chen, Shipeng Bai, Yue Han, Yabiao
Wang, Chengjie Wang, Yong Liu
- Abstract summary: Image harmonization aims to solve the visual inconsistency problem in composited images by adaptively adjusting the foreground pixels with the background as references.
Existing methods employ local color transformation or region matching between foreground and background, which neglects powerful proximity prior and independently distinguishes fore-/back-ground as a whole part for harmonization.
We propose a novel Global-aware Kernel Network (GKNet) to harmonize local regions with comprehensive consideration of long-distance background references.
- Score: 55.614444363743765
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Image harmonization aims to solve the visual inconsistency problem in
composited images by adaptively adjusting the foreground pixels with the
background as references. Existing methods employ local color transformation or
region matching between foreground and background, which neglects powerful
proximity prior and independently distinguishes fore-/back-ground as a whole
part for harmonization. As a result, they still show a limited performance
across varied foreground objects and scenes. To address this issue, we propose
a novel Global-aware Kernel Network (GKNet) to harmonize local regions with
comprehensive consideration of long-distance background references.
Specifically, GKNet includes two parts, \ie, harmony kernel prediction and
harmony kernel modulation branches. The former includes a Long-distance
Reference Extractor (LRE) to obtain long-distance context and Kernel Prediction
Blocks (KPB) to predict multi-level harmony kernels by fusing global
information with local features. To achieve this goal, a novel Selective
Correlation Fusion (SCF) module is proposed to better select relevant
long-distance background references for local harmonization. The latter employs
the predicted kernels to harmonize foreground regions with both local and
global awareness. Abundant experiments demonstrate the superiority of our
method for image harmonization over state-of-the-art methods, \eg, achieving
39.53dB PSNR that surpasses the best counterpart by +0.78dB $\uparrow$;
decreasing fMSE/MSE by 11.5\%$\downarrow$/6.7\%$\downarrow$ compared with the
SoTA method. Code will be available at
\href{https://github.com/XintianShen/GKNet}{here}.
Related papers
- Hierarchical Dynamic Image Harmonization [15.886047676987316]
We propose a hierarchical dynamic network (HDNet) to adapt features from local to global view for better feature transformation in efficient image harmonization.
The proposed HDNet significantly reduces the total model parameters by more than 80% compared to previous methods.
Notably, the HDNet achieves a 4% improvement in PSNR and a 19% reduction in MSE compared to the prior state-of-the-art methods.
arXiv Detail & Related papers (2022-11-16T03:15:19Z) - Image Harmonization with Region-wise Contrastive Learning [51.309905690367835]
We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme.
Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
arXiv Detail & Related papers (2022-05-27T15:46:55Z) - FRIH: Fine-grained Region-aware Image Harmonization [49.420765789360836]
We propose a novel global-local two stages framework for Fine-grained Region-aware Image Harmonization (FRIH)
Our algorithm achieves the best performance on iHarmony4 dataset (PSNR is 38.19 dB) with a lightweight model.
arXiv Detail & Related papers (2022-05-13T04:50:26Z) - Interactive Portrait Harmonization [99.15331091722231]
Current image harmonization methods consider the entire background as the guidance for harmonization.
A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed.
Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region.
arXiv Detail & Related papers (2022-03-15T19:30:34Z) - HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on
Heterogeneous Medical Images [19.62267284815759]
We introduce a new framework called HarmoFL to handle both local and global drifts.
HarmoFL mitigates the local update drift by normalizing amplitudes of images transformed into the frequency domain.
We show that HarmoFL outperforms a set of recent state-of-the-art methods with promising convergence behavior.
arXiv Detail & Related papers (2021-12-20T13:25:48Z) - Global Aggregation then Local Distribution for Scene Parsing [99.1095068574454]
We show that our approach can be modularized as an end-to-end trainable block and easily plugged into existing semantic segmentation networks.
Our approach allows us to build new state of the art on major semantic segmentation benchmarks including Cityscapes, ADE20K, Pascal Context, Camvid and COCO-stuff.
arXiv Detail & Related papers (2021-07-28T03:46:57Z) - Low Light Image Enhancement via Global and Local Context Modeling [164.85287246243956]
We introduce a context-aware deep network for low-light image enhancement.
First, it features a global context module that models spatial correlations to find complementary cues over full spatial domain.
Second, it introduces a dense residual block that captures local context with a relatively large receptive field.
arXiv Detail & Related papers (2021-01-04T09:40:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.