StyleDiT: A Unified Framework for Diverse Child and Partner Faces Synthesis with Style Latent Diffusion Transformer

• URL: http://arxiv.org/abs/2412.10785v1
• Date: Sat, 14 Dec 2024 10:47:17 GMT
• Title: StyleDiT: A Unified Framework for Diverse Child and Partner Faces Synthesis with Style Latent Diffusion Transformer
• Authors: Pin-Yen Chiu, Dai-Jie Wu, Po-Hsun Chu, Chia-Hsuan Hsu, Hsiang-Chen Chiu, Chih-Yu Wang, Jun-Cheng Chen,
• Abstract summary: StyleDiT is a novel framework that integrates the strengths of StyleGAN with the diffusion model to generate high-quality and diverse kinship faces. We introduce the Trait Guidance (RTG) mechanism, enabling independent control of influencing conditions. We extend the application to an unexplored domain: predicting a partner's facial images using a child's image and one parent's image.
• Score: 11.83733187403255
• License:
• Abstract: Kinship face synthesis is a challenging problem due to the scarcity and low quality of the available kinship data. Existing methods often struggle to generate descendants with both high diversity and fidelity while precisely controlling facial attributes such as age and gender. To address these issues, we propose the Style Latent Diffusion Transformer (StyleDiT), a novel framework that integrates the strengths of StyleGAN with the diffusion model to generate high-quality and diverse kinship faces. In this framework, the rich facial priors of StyleGAN enable fine-grained attribute control, while our conditional diffusion model is used to sample a StyleGAN latent aligned with the kinship relationship of conditioning images by utilizing the advantage of modeling complex kinship relationship distribution. StyleGAN then handles latent decoding for final face generation. Additionally, we introduce the Relational Trait Guidance (RTG) mechanism, enabling independent control of influencing conditions, such as each parent's facial image. RTG also enables a fine-grained adjustment between the diversity and fidelity in synthesized faces. Furthermore, we extend the application to an unexplored domain: predicting a partner's facial images using a child's image and one parent's image within the same framework. Extensive experiments demonstrate that our StyleDiT outperforms existing methods by striking an excellent balance between generating diverse and high-fidelity kinship faces.

Related papers

• HiFiVFS: High Fidelity Video Face Swapping [35.49571526968986]
  Face swapping aims to generate results that combine the identity from the source with attributes from the target. We propose a high fidelity video face swapping framework, which leverages the strong generative capability and temporal prior of Stable Video Diffusion. Our method achieves state-of-the-art (SOTA) in video face swapping, both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2024-11-27T12:30:24Z)
• Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models [69.50286698375386]
  We propose a novel approach that better harnesses diffusion models for face-swapping. We introduce a mask shuffling technique during inpainting training, which allows us to create a so-called universal model for swapping. Ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models.
  arXiv  Detail & Related papers  (2024-09-11T13:43:53Z)
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling [61.14140480095604]
  This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control.
  arXiv  Detail & Related papers  (2024-08-10T08:53:41Z)
• Controllable Face Synthesis with Semantic Latent Diffusion Models [6.438244172631555]
  We propose a SIS framework based on a novel Latent Diffusion Model architecture for human face generation and editing. The proposed system utilizes both SPADE normalization and cross-attention layers to merge shape and style information and, by doing so, allows for a precise control over each of the semantic parts of the human face.
  arXiv  Detail & Related papers  (2024-03-19T14:02:13Z)
• Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation [1.104121146441257]
  Deep generative models have shown impressive results in generating realistic images of faces. GANs managed to generate high-quality, high-fidelity images when conditioned on semantic masks, but they still lack the ability to diversify their output. We propose a multi-conditioning approach for diffusion models via cross-attention exploiting both attributes and semantic masks to generate high-quality and controllable face images.
  arXiv  Detail & Related papers  (2023-06-01T17:16:37Z)
• GANDiffFace: Controllable Generation of Synthetic Datasets for Face Recognition with Realistic Variations [2.7467281625529134]
  This study introduces GANDiffFace, a novel framework for the generation of synthetic datasets for face recognition. GANDiffFace combines the power of Generative Adversarial Networks (GANs) and Diffusion models to overcome the limitations of existing synthetic datasets.
  arXiv  Detail & Related papers  (2023-05-31T15:49:12Z)
• StyleSwap: Style-Based Generator Empowers Robust Face Swapping [90.05775519962303]
  We introduce a concise and effective framework named StyleSwap. Our core idea is to leverage a style-based generator to empower high-fidelity and robust face swapping. We identify that with only minimal modifications, a StyleGAN2 architecture can successfully handle the desired information from both source and target.
  arXiv  Detail & Related papers  (2022-09-27T16:35:16Z)
• Auto-regressive Image Synthesis with Integrated Quantization [55.51231796778219]
  This paper presents a versatile framework for conditional image generation. It incorporates the inductive bias of CNNs and powerful sequence modeling of auto-regression. Our method achieves superior diverse image generation performance as compared with the state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-21T22:19:17Z)
• High-resolution Face Swapping via Latent Semantics Disentanglement [50.23624681222619]
  We present a novel high-resolution hallucination face swapping method using the inherent prior knowledge of a pre-trained GAN model. We explicitly disentangle the latent semantics by utilizing the progressive nature of the generator. We extend our method to video face swapping by enforcing two-temporal constraints on the latent space and the image space.
  arXiv  Detail & Related papers  (2022-03-30T00:33:08Z)
• Cluster-guided Image Synthesis with Unconditional Models [41.89334167530054]
  This work focuses on controllable image generation by leveraging GANs that are well-trained in an unsupervised fashion. By conditioning on the cluster assignments, the proposed method is able to control the semantic class of the generated image. We showcase the efficacy of our approach on faces (CelebA-HQ and FFHQ), animals (Imagenet) and objects (LSUN) using different pre-trained generative models.
  arXiv  Detail & Related papers  (2021-12-24T02:18:34Z)
• InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs [73.27299786083424]
  We propose a framework called InterFaceGAN to interpret the disentangled face representation learned by state-of-the-art GAN models. We first find that GANs learn various semantics in some linear subspaces of the latent space. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection.
  arXiv  Detail & Related papers  (2020-05-18T18:01:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.