Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models

• URL: http://arxiv.org/abs/2503.21943v2
• Date: Mon, 07 Apr 2025 04:57:10 GMT
• Title: Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models
• Authors: Haoming Cai, Tsung-Wei Huang, Shiv Gehlot, Brandon Y. Feng, Sachin Shah, Guan-Ming Su, Christopher Metzler,
• Abstract summary: Shadow Director is a method that extracts and manipulates hidden shadow attributes within well-trained diffusion models.<n>Our approach uses a small estimation network that requires only a few thousand synthetic images and hours of training.
• Score: 7.954962037463368
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text-to-image diffusion models excel at generating diverse portraits, but lack intuitive shadow control. Existing editing approaches, as post-processing, struggle to offer effective manipulation across diverse styles. Additionally, these methods either rely on expensive real-world light-stage data collection or require extensive computational resources for training. To address these limitations, we introduce Shadow Director, a method that extracts and manipulates hidden shadow attributes within well-trained diffusion models. Our approach uses a small estimation network that requires only a few thousand synthetic images and hours of training-no costly real-world light-stage data needed. Shadow Director enables parametric and intuitive control over shadow shape, placement, and intensity during portrait generation while preserving artistic integrity and identity across diverse styles. Despite training only on synthetic data built on real-world identities, it generalizes effectively to generated portraits with diverse styles, making it a more accessible and resource-friendly solution.

Related papers

• Towards Hard and Soft Shadow Removal via Dual-Branch Separation Network and Vision Transformer [0.0]
  We propose a dual-path model that processes hard and soft shadows separately.<n>The model classifies shadow types and processes them through appropriate paths to produce shadow-free outputs.<n>Our model outperforms state-of-the-art methods and achieves 2.905 RMSE value on the ISTD dataset.
  arXiv  Detail & Related papers  (2025-01-03T15:29:46Z)
• Controllable Shadow Generation with Single-Step Diffusion Models from Synthetic Data [7.380444448047908]
  We introduce a novel method for fast, controllable, and background-free shadow generation for 2D object images.<n>We create a large synthetic dataset using a 3D rendering engine to train a diffusion model for controllable shadow generation.<n>We find that rectified flow objective achieves high-quality results with just a single sampling step enabling real-time applications.
  arXiv  Detail & Related papers  (2024-12-16T16:55:22Z)
• Generative Portrait Shadow Removal [27.98144439007323]
  We introduce a high-fidelity portrait shadow removal model that can effectively enhance the image of a portrait. Our method also demonstrates robustness to diverse subjects captured in real environments.
  arXiv  Detail & Related papers  (2024-10-07T22:09:22Z)
• Single-Image Shadow Removal Using Deep Learning: A Comprehensive Survey [78.84004293081631]
  The patterns of shadows are arbitrary, varied, and often have highly complex trace structures. The degradation caused by shadows is spatially non-uniform, resulting in inconsistencies in illumination and color between shadow and non-shadow areas. Recent developments in this field are primarily driven by deep learning-based solutions.
  arXiv  Detail & Related papers  (2024-07-11T20:58:38Z)
• LightIt: Illumination Modeling and Control for Diffusion Models [61.80461416451116]
  We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation. Our method is the first that enables the generation of images with controllable, consistent lighting.
  arXiv  Detail & Related papers  (2024-03-15T18:26:33Z)
• DiFaReli++: Diffusion Face Relighting with Consistent Cast Shadows [11.566896201650056]
  We introduce a novel approach to single-view face relighting in the wild, addressing challenges such as global illumination and cast shadows.<n>We propose a single-shot relighting framework that requires just one network pass, given pre-processed data, and even outperforms the teacher model across all metrics.
  arXiv  Detail & Related papers  (2023-04-19T08:03:20Z)
• Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation [76.96499178502759]
  Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting. Recent methods rely on deep learning to achieve high-quality results. We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
  arXiv  Detail & Related papers  (2022-09-21T17:15:58Z)
• Perspective Flow Aggregation for Data-Limited 6D Object Pose Estimation [121.02948087956955]
  For some applications, such as those in space or deep under water, acquiring real images, even unannotated, is virtually impossible. We propose a method that can be trained solely on synthetic images, or optionally using a few additional real images. It performs on par with methods that require annotated real images for training when not using any, and outperforms them considerably when using as few as twenty real images.
  arXiv  Detail & Related papers  (2022-03-18T10:20:21Z)
• Self-supervised Outdoor Scene Relighting [92.20785788740407]
  We propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.
  arXiv  Detail & Related papers  (2021-07-07T09:46:19Z)
• PhotoApp: Photorealistic Appearance Editing of Head Portraits [97.23638022484153]
  We present an approach for high-quality intuitive editing of the camera viewpoint and scene illumination in a portrait image. Most editing approaches rely on supervised learning using training data captured with setups such as light and camera stages. We design a supervised problem which learns in the latent space of StyleGAN. This combines the best of supervised learning and generative adversarial modeling.
  arXiv  Detail & Related papers  (2021-03-13T08:59:49Z)
• Portrait Shadow Manipulation [37.414681268753526]
  Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. We present a computational approach that gives casual photographers some of this control, thereby allowing poorly-lit portraits to be relit post-capture in a realistic and easily-controllable way. Our approach relies on a pair of neural networks---one to remove foreign shadows cast by external objects, and another to soften facial shadows cast by the features of the subject and to add a synthetic fill light to improve the lighting ratio.
  arXiv  Detail & Related papers  (2020-05-18T17:51:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.