PersonaHOI: Effortlessly Improving Personalized Face with Human-Object Interaction Generation

• URL: http://arxiv.org/abs/2501.05823v1
• Date: Fri, 10 Jan 2025 10:01:36 GMT
• Title: PersonaHOI: Effortlessly Improving Personalized Face with Human-Object Interaction Generation
• Abstract summary: PersonaHOI is a training- and tuning-free framework that fuses a general StableDiffusion model with a personalized face diffusion (PFD) model to generate identity-consistent human-object interaction (HOI) images.
• Score: 62.85437763951757
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce PersonaHOI, a training- and tuning-free framework that fuses a general StableDiffusion model with a personalized face diffusion (PFD) model to generate identity-consistent human-object interaction (HOI) images. While existing PFD models have advanced significantly, they often overemphasize facial features at the expense of full-body coherence, PersonaHOI introduces an additional StableDiffusion (SD) branch guided by HOI-oriented text inputs. By incorporating cross-attention constraints in the PFD branch and spatial merging at both latent and residual levels, PersonaHOI preserves personalized facial details while ensuring interactive non-facial regions. Experiments, validated by a novel interaction alignment metric, demonstrate the superior realism and scalability of PersonaHOI, establishing a new standard for practical personalized face with HOI generation. Our code will be available at https://github.com/JoyHuYY1412/PersonaHOI

Related papers

• Beyond Facial Consistency: Personalized Person Image Generation with Holistic Identity Preservation [31.445089759245146]
  We propose Dynamic Balancing Scaling (DBS) to improve face and appearance identity coordination.<n>DBS consists of two components: adaptive temporal gating, and region-aware optimization.<n>We also introduce Pexels-100, a benchmark for evaluating holistic identity consistency in personalized person generation.
  arXiv  Detail & Related papers  (2026-07-28T12:02:13Z)
• PACE: Persona Adaptation through Conversational Elicitation in Human-Robot Interaction [37.56295191425851]
  PACE is a novel framework for the interactive generation and deployment of structured personas on the Ameca humanoid robot.<n>Our system introduces an Interactive Persona Elicitation Pipeline, enabling the robot to synthesize a tailored, psychologically grounded identity.<n>We detail the Embodied System Integration required to translate this structured specification into expressive, multimodal humanoid behaviors.
  arXiv  Detail & Related papers  (2026-07-17T02:45:52Z)
• InsHuman: Towards Natural and Identity-Preserving Human Insertion [37.70158864105303]
  We propose InsHuman for natural and identity-preserving human insertion.<n> Specifically, we propose Human-Background Adaptive Fusion (HBAF), which detects foreground humans to obtain a binary mask.<n>We also propose Face-to-Face ID-Preserving (FFIP), which detects and matches faces between the generated image and the source image.
  arXiv  Detail & Related papers  (2026-05-08T07:58:03Z)
• DiffSwap++: 3D Latent-Controlled Diffusion for Identity-Preserving Face Swapping [16.846179110602737]
  We propose DiffSwap++, a novel diffusion-based face-swapping pipeline that incorporates 3D facial latent features during training.<n>Our method enhances geometric consistency and improves the disentanglement of facial identity from appearance attributes.<n>Experiments on CelebA, FFHQ, and CelebV-Text demonstrate that DiffSwap++ outperforms prior methods in preserving source identity while maintaining target pose and expression.
  arXiv  Detail & Related papers  (2025-11-04T18:56:49Z)
• Turn That Frown Upside Down: FaceID Customization via Cross-Training Data [49.51940625552275]
  CrossFaceID is the first large-scale, high-quality, and publicly available dataset designed to improve the facial modification capabilities of FaceID customization models. It consists of 40,000 text-image pairs from approximately 2,000 persons, with each person represented by around 20 images showcasing diverse facial attributes. During the training stage, a specific face of a person is used as input, and the FaceID customization model is forced to generate another image of the same person but with altered facial features. Experiments show that models fine-tuned on the CrossFaceID dataset its performance in preserving FaceID fidelity while significantly improving its
  arXiv  Detail & Related papers  (2025-01-26T05:27:38Z)
• PersonaCraft: Personalized and Controllable Full-Body Multi-Human Scene Generation Using Occlusion-Aware 3D-Conditioned Diffusion [11.86021588469762]
  PersonaCraft is a framework for controllable and occlusion-robust full-body personalized image synthesis of multiple individuals in complex scenes. Our dual-pathway body shape representation with SMPLx-based shape parameters and textual refinement, enables precise full-body personalization and flexible user-defined body shape adjustments.
  arXiv  Detail & Related papers  (2024-11-27T05:41:15Z)
• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration. We propose OSDFace, a novel one-step diffusion model for face restoration. Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• TCDiff: Triple Condition Diffusion Model with 3D Constraints for Stylizing Synthetic Faces [1.7535229154829601]
  Face recognition experiments using 1k, 2k, and 5k classes of our new dataset for training outperform state-of-the-art synthetic datasets in real face benchmarks.
  arXiv  Detail & Related papers  (2024-09-05T14:59:41Z)
• High-fidelity Person-centric Subject-to-Image Synthesis [13.785484396436367]
  Face-diffuser is an effective collaborative generation pipeline to eliminate the above training imbalance and quality compromise. The sampling process is divided into three sequential stages, i.e., semantic scene construction, subject-scene fusion, and subject enhancement. The subject-scene fusion stage, that is the collaboration achieved through a novel and highly effective mechanism, Saliency-adaptive Noise Fusion.
  arXiv  Detail & Related papers  (2023-11-17T05:03:53Z)
• FaceDancer: Pose- and Occlusion-Aware High Fidelity Face Swapping [62.38898610210771]
  We present a new single-stage method for subject face swapping and identity transfer, named FaceDancer. We have two major contributions: Adaptive Feature Fusion Attention (AFFA) and Interpreted Feature Similarity Regularization (IFSR)
  arXiv  Detail & Related papers  (2022-10-19T11:31:38Z)
• SuperFront: From Low-resolution to High-resolution Frontal Face Synthesis [65.35922024067551]
  We propose a generative adversarial network (GAN) -based model to generate high-quality, identity preserving frontal faces. Specifically, we propose SuperFront-GAN to synthesize a high-resolution (HR), frontal face from one-to-many LR faces with various poses. We integrate a super-resolution side-view module into SF-GAN to preserve identity information and fine details of the side-views in HR space.
  arXiv  Detail & Related papers  (2020-12-07T23:30:28Z)
• Dual-Attention GAN for Large-Pose Face Frontalization [59.689836951934694]
  We present a novel Dual-Attention Generative Adversarial Network (DA-GAN) for photo-realistic face frontalization. Specifically, a self-attention-based generator is introduced to integrate local features with their long-range dependencies. A novel face-attention-based discriminator is applied to emphasize local features of face regions.
  arXiv  Detail & Related papers  (2020-02-17T20:00:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.