Semi-supervised Parametric Real-world Image Harmonization

• URL: http://arxiv.org/abs/2303.00157v1
• Date: Wed, 1 Mar 2023 01:09:01 GMT
• Title: Semi-supervised Parametric Real-world Image Harmonization
• Authors: Ke Wang, Micha\"el Gharbi, He Zhang, Zhihao Xia and Eli Shechtman
• Abstract summary: Learning-based image harmonization techniques are usually trained to undo synthetic random global transformations applied to a masked foreground in a single ground truth photo. This simulated data does not model many of the important appearance mismatches between foreground and background in real composites. We propose a new semi-supervised training strategy that addresses this problem and lets us learn complex local appearance harmonization from unpaired real composites.
• Score: 38.66826236220168
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Learning-based image harmonization techniques are usually trained to undo synthetic random global transformations applied to a masked foreground in a single ground truth photo. This simulated data does not model many of the important appearance mismatches (illumination, object boundaries, etc.) between foreground and background in real composites, leading to models that do not generalize well and cannot model complex local changes. We propose a new semi-supervised training strategy that addresses this problem and lets us learn complex local appearance harmonization from unpaired real composites, where foreground and background come from different images. Our model is fully parametric. It uses RGB curves to correct the global colors and tone and a shading map to model local variations. Our method outperforms previous work on established benchmarks and real composites, as shown in a user study, and processes high-resolution images interactively.

Related papers

• Learning Dual-Level Deformable Implicit Representation for Real-World Scale Arbitrary Super-Resolution [81.74583887661794]
  We build a new real-world super-resolution benchmark with both integer and non-integer scaling factors. We propose a Dual-level Deformable Implicit Representation (DDIR) to solve real-world scale arbitrary super-resolution. Our trained model achieves state-of-the-art performance on the RealArbiSR and RealSR benchmarks for real-world scale arbitrary super-resolution.
  arXiv  Detail & Related papers  (2024-03-16T13:44:42Z)
• Intrinsic Harmonization for Illumination-Aware Compositing [0.7366405857677227]
  We introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain. First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region. A network then refines this inferred shading to generate a re-shading that aligns with the background scene.
  arXiv  Detail & Related papers  (2023-12-06T18:59:03Z)
• Deep Image Harmonization with Learnable Augmentation [17.690945824240348]
  Learnable augmentation is proposed to enrich the illumination diversity of small-scale datasets for better harmonization performance. SycoNet takes in a real image with foreground mask and a random vector to learn suitable color transformation, which is applied to the foreground of this real image to produce a synthetic composite image.
  arXiv  Detail & Related papers  (2023-08-01T08:40:23Z)
• CcHarmony: Color-checker based Image Harmonization Dataset [13.651034765401386]
  Image harmonization targets at adjusting the foreground in a composite image to make it compatible with the background, producing a more realistic and harmonious image. We construct an image harmonization dataset called ccHarmony, which is named after color checker (cc)
  arXiv  Detail & Related papers  (2022-06-01T23:57:16Z)
• FRIH: Fine-grained Region-aware Image Harmonization [49.420765789360836]
  We propose a novel global-local two stages framework for Fine-grained Region-aware Image Harmonization (FRIH) Our algorithm achieves the best performance on iHarmony4 dataset (PSNR is 38.19 dB) with a lightweight model.
  arXiv  Detail & Related papers  (2022-05-13T04:50:26Z)
• Image Harmonization by Matching Regional References [10.249228010611617]
  Recent image harmonization methods typically summarize the appearance pattern of global background and apply it to the global foreground without location discrepancy. For a real image, the appearances (illumination, color temperature, saturation, hue, texture, etc) of different regions can vary significantly. Previous methods, which transfer the appearance globally, are not optimal.
  arXiv  Detail & Related papers  (2022-04-10T16:23:06Z)
• SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
  We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
  arXiv  Detail & Related papers  (2021-08-15T19:51:33Z)
• SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses [82.56853587380168]
  We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same. We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters. Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T08:23:25Z)
• Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition [67.9464567157846]
  We propose an autoencoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties. Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-06-29T12:53:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.