Human Multi-View Synthesis from a Single-View Model:Transferred Body and Face Representations

• URL: http://arxiv.org/abs/2412.03011v1
• Date: Wed, 04 Dec 2024 04:02:17 GMT
• Title: Human Multi-View Synthesis from a Single-View Model:Transferred Body and Face Representations
• Authors: Yu Feng, Shunsi Zhang, Jian Shu, Hanfeng Zhao, Guoliang Pang, Chi Zhang, Hao Wang,
• Abstract summary: We propose an innovative framework that leverages transferred body and facial representations for multi-view human synthesis.<n> Specifically, we use a single-view model pretrained on a large-scale human dataset to develop a multi-view body representation.<n>Our approach outperforms the current state-of-the-art methods, achieving superior performance in multi-view human synthesis.
• Score: 7.448124739584319
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating multi-view human images from a single view is a complex and significant challenge. Although recent advancements in multi-view object generation have shown impressive results with diffusion models, novel view synthesis for humans remains constrained by the limited availability of 3D human datasets. Consequently, many existing models struggle to produce realistic human body shapes or capture fine-grained facial details accurately. To address these issues, we propose an innovative framework that leverages transferred body and facial representations for multi-view human synthesis. Specifically, we use a single-view model pretrained on a large-scale human dataset to develop a multi-view body representation, aiming to extend the 2D knowledge of the single-view model to a multi-view diffusion model. Additionally, to enhance the model's detail restoration capability, we integrate transferred multimodal facial features into our trained human diffusion model. Experimental evaluations on benchmark datasets demonstrate that our approach outperforms the current state-of-the-art methods, achieving superior performance in multi-view human synthesis.

Related papers

• Multi-identity Human Image Animation with Structural Video Diffusion [64.20452431561436]
  We present Structural Video Diffusion, a novel framework for generating realistic multi-human videos. Our approach introduces identity-specific embeddings to maintain consistent appearances across individuals. We expand existing human video dataset with 25K new videos featuring diverse multi-human and object interaction scenarios.
  arXiv  Detail & Related papers  (2025-04-05T10:03:49Z)
• Zero-Shot Human-Object Interaction Synthesis with Multimodal Priors [31.277540988829976]
  This paper proposes a novel zero-shot HOI synthesis framework without relying on end-to-end training on currently limited 3D HOI datasets. We employ pre-trained human pose estimation models to extract human poses and introduce a generalizable category-level 6-DoF estimation method to obtain the object poses from 2D HOI images.
  arXiv  Detail & Related papers  (2025-03-25T23:55:47Z)
• MEAT: Multiview Diffusion Model for Human Generation on Megapixels with Mesh Attention [83.56588173102594]
  We introduce a solution called mesh attention to enable training at 1024x1024 resolution. This approach significantly reduces the complexity of multiview attention while maintaining cross-view consistency. Building on this foundation, we devise a mesh attention block and combine it with keypoint conditioning to create our human-specific multiview diffusion model, MEAT.
  arXiv  Detail & Related papers  (2025-03-11T17:50:59Z)
• HumanGif: Single-View Human Diffusion with Generative Prior [25.516544735593087]
  We propose HumanGif, a single-view human diffusion model with generative priors. Specifically, we formulate the single-view-based 3D human novel view and pose synthesis as a single-view-conditioned human diffusion process. We show that HumanGif achieves the best perceptual performance, with better generalizability for novel view and pose synthesis.
  arXiv  Detail & Related papers  (2025-02-17T17:55:27Z)
• PSHuman: Photorealistic Single-view Human Reconstruction using Cross-Scale Diffusion [43.850899288337025]
  PSHuman is a novel framework that explicitly reconstructs human meshes utilizing priors from the multiview diffusion model. It is found that directly applying multiview diffusion on single-view human images leads to severe geometric distortions. To enhance cross-view body shape consistency of varied human poses, we condition the generative model on parametric models like SMPL-X.
  arXiv  Detail & Related papers  (2024-09-16T10:13:06Z)
• Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM [29.13412037370585]
  We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image. Our method is able to capture human without any template prior, e.g., SMPL, and effectively enhance occluded parts with rich and realistic details.
  arXiv  Detail & Related papers  (2024-01-22T18:08:22Z)
• Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation [14.064983137553353]
  We aim to enhance the quality and functionality of generative diffusion models for the task of creating controllable, photorealistic human avatars. We achieve this by integrating a 3D morphable model into the state-of-the-art multi-view-consistent diffusion approach. Our proposed framework is the first diffusion model to enable the creation of fully 3D-consistent, animatable, and photorealistic human avatars.
  arXiv  Detail & Related papers  (2024-01-09T18:59:04Z)
• InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation [61.62346472443454]
  InceptionHuman is a prompt-to-NeRF framework that allows easy control via a combination of prompts in different modalities to generate photorealistic 3D humans. InceptionHuman achieves consistent 3D human generation within a progressively refined NeRF space.
  arXiv  Detail & Related papers  (2023-11-27T15:49:41Z)
• SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling [93.60731530276911]
  We introduce a new synthetic dataset, SynBody, with three appealing features. The dataset comprises 1.2M images with corresponding accurate 3D annotations, covering 10,000 human body models, 1,187 actions, and various viewpoints.
  arXiv  Detail & Related papers  (2023-03-30T13:30:12Z)
• Human Image Generation: A Comprehensive Survey [44.204029557298476]
  In this paper, we divide human image generation techniques into three paradigms, i.e., data-driven methods, knowledge-guided methods and hybrid methods. The advantages and characteristics of different methods are summarized in terms of model architectures. Due to the wide application potentials, the typical downstream usages of synthesized human images are covered.
  arXiv  Detail & Related papers  (2022-12-17T15:19:45Z)
• Progressive Multi-view Human Mesh Recovery with Self-Supervision [68.60019434498703]
  Existing solutions typically suffer from poor generalization performance to new settings. We propose a novel simulation-based training pipeline for multi-view human mesh recovery.
  arXiv  Detail & Related papers  (2022-12-10T06:28:29Z)
• Human Performance Capture from Monocular Video in the Wild [50.34917313325813]
  We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW.
  arXiv  Detail & Related papers  (2021-11-29T16:32:41Z)
• HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge [33.26419876973344]
  This paper presents a new large multiview dataset called HUMBI for human body expressions with natural clothing. 107 synchronized HD cameras are used to capture 772 distinctive subjects across gender, ethnicity, age, and style. We reconstruct high fidelity body expressions using 3D mesh models, which allows representing view-specific appearance.
  arXiv  Detail & Related papers  (2021-09-30T23:19:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.