Diffusion Models are Efficient Data Generators for Human Mesh Recovery

• URL: http://arxiv.org/abs/2403.11111v3
• Date: Fri, 17 Oct 2025 07:56:49 GMT
• Title: Diffusion Models are Efficient Data Generators for Human Mesh Recovery
• Authors: Yongtao Ge, Wenjia Wang, Yongfan Chen, Fanzhou Wang, Lei Yang, Hao Chen, Chunhua Shen,
• Abstract summary: We show that synthetic data created by generative models is complementary to CG-rendered data.<n>We propose an effective data generation pipeline based on recent diffusion models, termed HumanWild.<n>Our work could pave the way for scaling up 3D human recovery to in-the-wild scenes.
• Score: 55.37787289869703
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite remarkable progress having been made on the problem of 3D human pose and shape estimation (HPS), current state-of-the-art methods rely heavily on either confined indoor mocap datasets or datasets generated by a rendering engine using computer graphics (CG). Both categories of datasets exhibit inadequacies in furnishing adequate human identities and authentic in-the-wild background scenes, which are crucial for accurately simulating real-world distributions. In this work, we show that synthetic data created by generative models is complementary to CG-rendered data for achieving remarkable generalization performance on diverse real-world scenes. We propose an effective data generation pipeline based on recent diffusion models, termed HumanWild, which can effortlessly generate human images and corresponding 3D mesh annotations. Specifically, we first collect a large-scale human-centric dataset with comprehensive annotations, e.g, text captions, the depth map, and surface normal images. To generate a wide variety of human images with initial labels, we train a customized, multi-condition ControlNet model. The key to this process is using a 3D parametric model, e.g, SMPL-X, to create various condition inputs easily. Our data generation pipeline is both flexible and customizable, making it adaptable to multiple real-world tasks, such as human interaction in complex scenes and humans captured by wide-angle lenses. By relying solely on generative models, we can produce large-scale, in-the-wild human images with high-quality annotations, significantly reducing the need for manual image collection and annotation. The generated dataset encompasses a wide range of viewpoints, environments, and human identities, ensuring its versatility across different scenarios. We hope that our work could pave the way for scaling up 3D human recovery to in-the-wild scenes.

Related papers

• HuGeDiff: 3D Human Generation via Diffusion with Gaussian Splatting [33.9893684177763]
  Current methods struggle with fine detail, accurate rendering of hands and faces, human realism, and controlability over appearance.<n>We present a weakly supervised pipeline that tries to address these challenges.<n>We demonstrate orders-of-magnitude speed-ups in 3D human generation compared to the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-06-04T18:11:23Z)
• SIGMAN:Scaling 3D Human Gaussian Generation with Millions of Assets [72.26350984924129]
  We propose a latent space generation paradigm for 3D human digitization. We transform the ill-posed low-to-high-dimensional mapping problem into a learnable distribution shift. We employ the multi-view optimization approach combined with synthetic data to construct the HGS-1M dataset.
  arXiv  Detail & Related papers  (2025-04-09T15:38:18Z)
• Zero-Shot Human-Object Interaction Synthesis with Multimodal Priors [31.277540988829976]
  This paper proposes a novel zero-shot HOI synthesis framework without relying on end-to-end training on currently limited 3D HOI datasets.<n>We employ pre-trained human pose estimation models to extract human poses and introduce a generalizable category-level 6-DoF estimation method to obtain the object poses from 2D HOI images.
  arXiv  Detail & Related papers  (2025-03-25T23:55:47Z)
• FAMOUS: High-Fidelity Monocular 3D Human Digitization Using View Synthesis [51.193297565630886]
  The challenge of accurately inferring texture remains, particularly in obscured areas such as the back of a person in frontal-view images. This limitation in texture prediction largely stems from the scarcity of large-scale and diverse 3D datasets. We propose leveraging extensive 2D fashion datasets to enhance both texture and shape prediction in 3D human digitization.
  arXiv  Detail & Related papers  (2024-10-13T01:25:05Z)
• 3D-VirtFusion: Synthetic 3D Data Augmentation through Generative Diffusion Models and Controllable Editing [52.68314936128752]
  We propose a new paradigm to automatically generate 3D labeled training data by harnessing the power of pretrained large foundation models. For each target semantic class, we first generate 2D images of a single object in various structure and appearance via diffusion models and chatGPT generated text prompts. We transform these augmented images into 3D objects and construct virtual scenes by random composition.
  arXiv  Detail & Related papers  (2024-08-25T09:31:22Z)
• Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation [32.30055363306321]
  We propose a paradigm for seamlessly unifying different human pose and shape-related tasks and datasets. Our formulation is centered on the ability - both at training and test time - to query any arbitrary point of the human volume. We can naturally exploit differently annotated data sources including mesh, 2D/3D skeleton and dense pose, without having to convert between them.
  arXiv  Detail & Related papers  (2024-07-10T10:44:18Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• GeoGen: Geometry-Aware Generative Modeling via Signed Distance Functions [22.077366472693395]
  We introduce a new generative approach for synthesizing 3D geometry and images from single-view collections. By employing volumetric rendering using neural radiance fields, they inherit a key limitation: the generated geometry is noisy and unconstrained. We propose GeoGen, a new SDF-based 3D generative model trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2024-06-06T17:00:10Z)
• ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models [65.22994156658918]
  We present a method that learns to generate multi-view images in a single denoising process from real-world data. We design an autoregressive generation that renders more 3D-consistent images at any viewpoint.
  arXiv  Detail & Related papers  (2024-03-04T07:57:05Z)
• Learning Dense Correspondence from Synthetic Environments [27.841736037738286]
  Existing methods map manually labelled human pixels in real 2D images onto the 3D surface, which is prone to human error. We propose to solve the problem of data scarcity by training 2D-3D human mapping algorithms using automatically generated synthetic data.
  arXiv  Detail & Related papers  (2022-03-24T08:13:26Z)
• 3D-Aware Semantic-Guided Generative Model for Human Synthesis [67.86621343494998]
  This paper proposes a 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis. Our experiments on the DeepFashion dataset show that 3D-SGAN significantly outperforms the most recent baselines.
  arXiv  Detail & Related papers  (2021-12-02T17:10:53Z)
• UltraPose: Synthesizing Dense Pose with 1 Billion Points by Human-body Decoupling 3D Model [58.70130563417079]
  We introduce a new 3D human-body model with a series of decoupled parameters that could freely control the generation of the body. Compared to the existing manually annotated DensePose-COCO dataset, the synthetic UltraPose has ultra dense image-to-surface correspondences without annotation cost and error.
  arXiv  Detail & Related papers  (2021-10-28T16:24:55Z)
• Methodology for Building Synthetic Datasets with Virtual Humans [1.5556923898855324]
  Large datasets can be used for improved, targeted training of deep neural networks. In particular, we make use of a 3D morphable face model for the rendering of multiple 2D images across a dataset of 100 synthetic identities.
  arXiv  Detail & Related papers  (2020-06-21T10:29:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.