DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery

• URL: http://arxiv.org/abs/2404.01424v1
• Date: Mon, 1 Apr 2024 18:59:13 GMT
• Title: DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery
• Authors: Yixuan Zhu, Ao Li, Yansong Tang, Wenliang Zhao, Jie Zhou, Jiwen Lu,
• Abstract summary: DPMesh is an innovative framework for occluded human mesh recovery. It capitalizes on the profound diffusion prior about object structure and spatial relationships embedded in a pre-trained text-to-image diffusion model.
• Score: 71.6345505427213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recovery of occluded human meshes presents challenges for current methods due to the difficulty in extracting effective image features under severe occlusion. In this paper, we introduce DPMesh, an innovative framework for occluded human mesh recovery that capitalizes on the profound diffusion prior about object structure and spatial relationships embedded in a pre-trained text-to-image diffusion model. Unlike previous methods reliant on conventional backbones for vanilla feature extraction, DPMesh seamlessly integrates the pre-trained denoising U-Net with potent knowledge as its image backbone and performs a single-step inference to provide occlusion-aware information. To enhance the perception capability for occluded poses, DPMesh incorporates well-designed guidance via condition injection, which produces effective controls from 2D observations for the denoising U-Net. Furthermore, we explore a dedicated noisy key-point reasoning approach to mitigate disturbances arising from occlusion and crowded scenarios. This strategy fully unleashes the perceptual capability of the diffusion prior, thereby enhancing accuracy. Extensive experiments affirm the efficacy of our framework, as we outperform state-of-the-art methods on both occlusion-specific and standard datasets. The persuasive results underscore its ability to achieve precise and robust 3D human mesh recovery, particularly in challenging scenarios involving occlusion and crowded scenes.

Related papers

• Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration [64.84134880709625]
  We show that it is possible to perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful diffusion loss. We present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model.
  arXiv  Detail & Related papers  (2024-06-26T17:40:30Z)
• Digging into contrastive learning for robust depth estimation with diffusion models [55.62276027922499]
  We propose a novel robust depth estimation method called D4RD. It features a custom contrastive learning mode tailored for diffusion models to mitigate performance degradation in complex environments. In experiments, D4RD surpasses existing state-of-the-art solutions on synthetic corruption datasets and real-world weather conditions.
  arXiv  Detail & Related papers  (2024-04-15T14:29:47Z)
• DMAT: A Dynamic Mask-Aware Transformer for Human De-occlusion [5.901982216097867]
  Human de-occlusion aims to infer the appearance of invisible human parts from an occluded image. This paper proposes a dynamic mask-aware transformer (DMAT), which dynamically augments information from human regions. Experiments on the AHP dataset demonstrate its superior performance compared to recent state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-07T03:36:41Z)
• A comprehensive framework for occluded human pose estimation [10.92234109536279]
  Occlusion presents a significant challenge in human pose estimation. We propose DAG (Data, Attention, Graph) to address the performance degradation caused by occluded human pose estimation. We also present the Feature-Guided Multi-Hop GCN (FGMP-GCN) to fully explore the prior knowledge of body structure and improve pose estimation results.
  arXiv  Detail & Related papers  (2023-12-30T06:55:30Z)
• Distribution-Aligned Diffusion for Human Mesh Recovery [16.64567393672489]
  We propose a diffusion-based approach for human mesh recovery. We propose a Human Mesh Diffusion (HMDiff) framework which frames mesh recovery as a reverse diffusion process. Our method achieves state-of-the-art performance on three widely used datasets.
  arXiv  Detail & Related papers  (2023-08-25T13:29:31Z)
• CamoDiffusion: Camouflaged Object Detection via Conditional Diffusion Models [72.93652777646233]
  Camouflaged Object Detection (COD) is a challenging task in computer vision due to the high similarity between camouflaged objects and their surroundings. We propose a new paradigm that treats COD as a conditional mask-generation task leveraging diffusion models. Our method, dubbed CamoDiffusion, employs the denoising process of diffusion models to iteratively reduce the noise of the mask.
  arXiv  Detail & Related papers  (2023-05-29T07:49:44Z)
• The role of noise in denoising models for anomaly detection in medical images [62.0532151156057]
  Pathological brain lesions exhibit diverse appearance in brain images. Unsupervised anomaly detection approaches have been proposed using only normal data for training. We show that optimization of the spatial resolution and magnitude of the noise improves the performance of different model training regimes.
  arXiv  Detail & Related papers  (2023-01-19T21:39:38Z)
• Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation [33.86986028882488]
  Occlusion poses a great threat to monocular multi-person 3D human pose estimation due to large variability in terms of the shape, appearance, and position of occluders. Existing methods try to handle occlusion with pose priors/constraints, data augmentation, or implicit reasoning. We develop a method to explicitly model this process that significantly improves bottom-up multi-person human pose estimation.
  arXiv  Detail & Related papers  (2022-07-29T22:12:50Z)
• Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimation [70.32536356351706]
  We introduce MRP-Net that constitutes a common deep network backbone with two output heads subscribing to two diverse configurations. We derive suitable measures to quantify prediction uncertainty at both pose and joint level. We present a comprehensive evaluation of the proposed approach and demonstrate state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-29T07:14:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.