Intrinsic Harmonization for Illumination-Aware Compositing
- URL: http://arxiv.org/abs/2312.03698v2
- Date: Thu, 7 Dec 2023 02:19:27 GMT
- Title: Intrinsic Harmonization for Illumination-Aware Compositing
- Authors: Chris Careaga, S. Mahdi H. Miangoleh, Ya\u{g}{\i}z Aksoy
- Abstract summary: We introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain.
First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region.
A network then refines this inferred shading to generate a re-shading that aligns with the background scene.
- Score: 0.7366405857677227
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Despite significant advancements in network-based image harmonization
techniques, there still exists a domain disparity between typical training
pairs and real-world composites encountered during inference. Most existing
methods are trained to reverse global edits made on segmented image regions,
which fail to accurately capture the lighting inconsistencies between the
foreground and background found in composited images. In this work, we
introduce a self-supervised illumination harmonization approach formulated in
the intrinsic image domain. First, we estimate a simple global lighting model
from mid-level vision representations to generate a rough shading for the
foreground region. A network then refines this inferred shading to generate a
harmonious re-shading that aligns with the background scene. In order to match
the color appearance of the foreground and background, we utilize ideas from
prior harmonization approaches to perform parameterized image edits in the
albedo domain. To validate the effectiveness of our approach, we present
results from challenging real-world composites and conduct a user study to
objectively measure the enhanced realism achieved compared to state-of-the-art
harmonization methods.
Related papers
- Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration [64.84134880709625]
We show that it is possible to perform domain adaptation via the noise space using diffusion models.
In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful diffusion loss.
We present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model.
arXiv Detail & Related papers (2024-06-26T17:40:30Z) - Relightful Harmonization: Lighting-aware Portrait Background Replacement [23.19641174787912]
We introduce Relightful Harmonization, a lighting-aware diffusion model designed to seamlessly harmonize sophisticated lighting effect for the foreground portrait using any background image.
Our approach unfolds in three stages. First, we introduce a lighting representation module that allows our diffusion model to encode lighting information from target image background.
Second, we introduce an alignment network that aligns lighting features learned from image background with lighting features learned from panorama environment maps.
arXiv Detail & Related papers (2023-12-11T23:20:31Z) - Image Harmonization with Diffusion Model [26.183879349798588]
Inconsistent lighting conditions between the foreground and background often result in unrealistic composites.
We present a novel approach for image harmonization by leveraging diffusion models.
arXiv Detail & Related papers (2023-06-17T23:23:52Z) - Semi-supervised Parametric Real-world Image Harmonization [38.66826236220168]
Learning-based image harmonization techniques are usually trained to undo synthetic random global transformations applied to a masked foreground in a single ground truth photo.
This simulated data does not model many of the important appearance mismatches between foreground and background in real composites.
We propose a new semi-supervised training strategy that addresses this problem and lets us learn complex local appearance harmonization from unpaired real composites.
arXiv Detail & Related papers (2023-03-01T01:09:01Z) - Image Harmonization with Region-wise Contrastive Learning [51.309905690367835]
We propose a novel image harmonization framework with external style fusion and region-wise contrastive learning scheme.
Our method attempts to bring together corresponding positive and negative samples by maximizing the mutual information between the foreground and background styles.
arXiv Detail & Related papers (2022-05-27T15:46:55Z) - FRIH: Fine-grained Region-aware Image Harmonization [49.420765789360836]
We propose a novel global-local two stages framework for Fine-grained Region-aware Image Harmonization (FRIH)
Our algorithm achieves the best performance on iHarmony4 dataset (PSNR is 38.19 dB) with a lightweight model.
arXiv Detail & Related papers (2022-05-13T04:50:26Z) - SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited.
Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
arXiv Detail & Related papers (2021-08-15T19:51:33Z) - BargainNet: Background-Guided Domain Translation for Image Harmonization [26.370523451625466]
Unharmonious foreground and background downgrade the quality of composite image.
Image harmonization, which adjusts the foreground to improve the consistency, is an essential yet challenging task.
We propose an image harmonization network with a novel domain code extractor and well-tailored triplet losses.
arXiv Detail & Related papers (2020-09-19T05:14:08Z) - Adversarial Image Composition with Auxiliary Illumination [53.89445873577062]
We propose an Adversarial Image Composition Net (AIC-Net) that achieves realistic image composition.
A novel branched generation mechanism is proposed, which disentangles the generation of shadows and the transfer of foreground styles.
Experiments on pedestrian and car composition tasks show that the proposed AIC-Net achieves superior composition performance.
arXiv Detail & Related papers (2020-09-17T12:58:16Z) - Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement [78.58603635621591]
Training an unpaired synthetic-to-real translation network in image space is severely under-constrained.
We propose a semi-supervised approach that operates on the disentangled shading and albedo layers of the image.
Our two-stage pipeline first learns to predict accurate shading in a supervised fashion using physically-based renderings as targets.
arXiv Detail & Related papers (2020-03-27T21:45:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.