Person-In-Situ: Scene-Consistent Human Image Insertion with Occlusion-Aware Pose Control

• URL: http://arxiv.org/abs/2505.04052v1
• Date: Wed, 07 May 2025 01:47:15 GMT
• Title: Person-In-Situ: Scene-Consistent Human Image Insertion with Occlusion-Aware Pose Control
• Authors: Shun Masuda, Yuki Endo, Yoshihiro Kanamori,
• Abstract summary: Existing methods can't handle occlusion of the inserted person by foreground objects and unnaturally place the person in the frontmost layer.<n>We propose two methods to address these challenges.<n>Both allow explicit pose control via a 3D body model and leverage latent diffusion models to synthesize the person at a contextually appropriate depth.
• Score: 1.529342790344802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compositing human figures into scene images has broad applications in areas such as entertainment and advertising. However, existing methods often cannot handle occlusion of the inserted person by foreground objects and unnaturally place the person in the frontmost layer. Moreover, they offer limited control over the inserted person's pose. To address these challenges, we propose two methods. Both allow explicit pose control via a 3D body model and leverage latent diffusion models to synthesize the person at a contextually appropriate depth, naturally handling occlusions without requiring occlusion masks. The first is a two-stage approach: the model first learns a depth map of the scene with the person through supervised learning, and then synthesizes the person accordingly. The second method learns occlusion implicitly and synthesizes the person directly from input data without explicit depth supervision. Quantitative and qualitative evaluations show that both methods outperform existing approaches by better preserving scene consistency while accurately reflecting occlusions and user-specified poses.

Related papers

• PersonaCraft: Personalized and Controllable Full-Body Multi-Human Scene Generation Using Occlusion-Aware 3D-Conditioned Diffusion [11.86021588469762]
  PersonaCraft is a framework for controllable and occlusion-robust full-body personalized image synthesis of multiple individuals in complex scenes.<n>Our dual-pathway body shape representation with SMPLx-based shape parameters and textual refinement, enables precise full-body personalization and flexible user-defined body shape adjustments.
  arXiv  Detail & Related papers  (2024-11-27T05:41:15Z)
• Text2Place: Affordance-aware Text Guided Human Placement [26.041917073228483]
  This work tackles the problem of realistic human insertion in a given background scene termed as textbfSemantic Human Placement. For learning semantic masks, we leverage rich object-scene priors learned from the text-to-image generative models. The proposed method can generate highly realistic scene compositions while preserving the background and subject identity.
  arXiv  Detail & Related papers  (2024-07-22T08:00:06Z)
• MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion [22.62170098534097]
  We propose MagicPose, a diffusion-based model for 2D human pose and facial expression. By leveraging the prior knowledge of image diffusion models, MagicPose generalizes well to unseen human identities and complex poses. The proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion.
  arXiv  Detail & Related papers  (2023-11-18T10:22:44Z)
• Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation [72.50214227616728]
  Several methods have proposed to learn image representations in a self-supervised fashion so as to disentangle the appearance information from the pose one. We study disentanglement from the perspective of the self-supervised network, via diverse image synthesis experiments. We design an adversarial strategy focusing on generating natural appearance changes of the subject, and against which we could expect a disentangled network to be robust.
  arXiv  Detail & Related papers  (2023-09-20T22:22:21Z)
• Semantically Consistent Person Image Generation [18.73832646369506]
  We propose a data-driven approach for context-aware person image generation.<n>In our method, the position, scale, and appearance of the generated person are semantically conditioned on the existing persons in the scene.
  arXiv  Detail & Related papers  (2023-02-28T16:34:55Z)
• OccluMix: Towards De-Occlusion Virtual Try-on by Semantically-Guided Mixup [79.3118064406151]
  Image Virtual try-on aims at replacing the cloth on a personal image with a garment image (in-shop clothes) Prior methods successfully preserve the character of clothing images. Occlusion remains a pernicious effect for realistic virtual try-on.
  arXiv  Detail & Related papers  (2023-01-03T06:29:11Z)
• Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation [33.86986028882488]
  Occlusion poses a great threat to monocular multi-person 3D human pose estimation due to large variability in terms of the shape, appearance, and position of occluders. Existing methods try to handle occlusion with pose priors/constraints, data augmentation, or implicit reasoning. We develop a method to explicitly model this process that significantly improves bottom-up multi-person human pose estimation.
  arXiv  Detail & Related papers  (2022-07-29T22:12:50Z)
• Unsupervised Object Learning via Common Fate [61.14802390241075]
  Learning generative object models from unlabelled videos is a long standing problem and required for causal scene modeling. We decompose this problem into three easier subtasks, and provide candidate solutions for each of them. We show that our approach allows learning generative models that generalize beyond the occlusions present in the input videos.
  arXiv  Detail & Related papers  (2021-10-13T08:22:04Z)
• Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency [55.94908688207493]
  We propose a self-supervised framework named SPICE that closes the image quality gap with supervised methods. The key insight enabling self-supervision is to exploit 3D information about the human body in several ways. SPICE achieves state-of-the-art performance on the DeepFashion dataset.
  arXiv  Detail & Related papers  (2021-10-11T17:48:50Z)
• Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement [63.853412753242615]
  Learning a good 3D human pose representation is important for human pose related tasks. We propose a novel Siamese denoising autoencoder to learn a 3D pose representation. Our approach achieves state-of-the-art performance on two inherently different tasks.
  arXiv  Detail & Related papers  (2020-07-14T14:25:22Z)
• Wish You Were Here: Context-Aware Human Generation [100.51309746913512]
  We present a novel method for inserting objects, specifically humans, into existing images. Our method involves threeworks: the first generates the semantic map of the new person, given the pose of the other persons in the scene. The second network renders the pixels of the novel person and its blending mask, based on specifications in the form of multiple appearance components. A third network refines the generated face in order to match those of the target person.
  arXiv  Detail & Related papers  (2020-05-21T14:09:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.