DPHMs: Diffusion Parametric Head Models for Depth-based Tracking
- URL: http://arxiv.org/abs/2312.01068v2
- Date: Mon, 8 Apr 2024 14:33:12 GMT
- Title: DPHMs: Diffusion Parametric Head Models for Depth-based Tracking
- Authors: Jiapeng Tang, Angela Dai, Yinyu Nie, Lev Markhasin, Justus Thies, Matthias Niessner,
- Abstract summary: We introduce Diffusion Parametric Head Models (DPHMs)
DPHMs are a generative model that enables robust volumetric head reconstruction and tracking from monocular depth sequences.
We propose a latent diffusion-based prior to regularize volumetric head reconstruction and tracking.
- Score: 42.016598097736626
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We introduce Diffusion Parametric Head Models (DPHMs), a generative model that enables robust volumetric head reconstruction and tracking from monocular depth sequences. While recent volumetric head models, such as NPHMs, can now excel in representing high-fidelity head geometries, tracking and reconstructing heads from real-world single-view depth sequences remains very challenging, as the fitting to partial and noisy observations is underconstrained. To tackle these challenges, we propose a latent diffusion-based prior to regularize volumetric head reconstruction and tracking. This prior-based regularizer effectively constrains the identity and expression codes to lie on the underlying latent manifold which represents plausible head shapes. To evaluate the effectiveness of the diffusion-based prior, we collect a dataset of monocular Kinect sequences consisting of various complex facial expression motions and rapid transitions. We compare our method to state-of-the-art tracking methods and demonstrate improved head identity reconstruction as well as robust expression tracking.
Related papers
- Stratified Avatar Generation from Sparse Observations [10.291918304187769]
Estimating 3D full-body avatars from AR/VR devices is essential for creating immersive experiences.
In this paper, we are inspired by the inherent property of the kinematic tree defined in the Skinned Multi-Person Linear (SMPL) model.
We propose a stratified approach to decouple the conventional full-body avatar reconstruction pipeline into two stages.
arXiv Detail & Related papers (2024-05-30T06:25:42Z) - HeadRecon: High-Fidelity 3D Head Reconstruction from Monocular Video [37.53752896927615]
We study the reconstruction of high-fidelity 3D head models from arbitrary monocular videos.
We propose a prior-guided dynamic implicit neural network to tackle these problems.
arXiv Detail & Related papers (2023-12-14T12:38:56Z) - Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust.
Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model.
We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
arXiv Detail & Related papers (2023-07-31T10:22:33Z) - CamoDiffusion: Camouflaged Object Detection via Conditional Diffusion
Models [72.93652777646233]
Camouflaged Object Detection (COD) is a challenging task in computer vision due to the high similarity between camouflaged objects and their surroundings.
We propose a new paradigm that treats COD as a conditional mask-generation task leveraging diffusion models.
Our method, dubbed CamoDiffusion, employs the denoising process of diffusion models to iteratively reduce the noise of the mask.
arXiv Detail & Related papers (2023-05-29T07:49:44Z) - Hierarchical Integration Diffusion Model for Realistic Image Deblurring [71.76410266003917]
Diffusion models (DMs) have been introduced in image deblurring and exhibited promising performance.
We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring.
Experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.
arXiv Detail & Related papers (2023-05-22T12:18:20Z) - Free-HeadGAN: Neural Talking Head Synthesis with Explicit Gaze Control [54.079327030892244]
Free-HeadGAN is a person-generic neural talking head synthesis system.
We show that modeling faces with sparse 3D facial landmarks are sufficient for achieving state-of-the-art generative performance.
arXiv Detail & Related papers (2022-08-03T16:46:08Z) - MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface
Reconstruction [72.05649682685197]
State-of-the-art neural implicit methods allow for high-quality reconstructions of simple scenes from many input views.
This is caused primarily by the inherent ambiguity in the RGB reconstruction loss that does not provide enough constraints.
Motivated by recent advances in the area of monocular geometry prediction, we explore the utility these cues provide for improving neural implicit surface reconstruction.
arXiv Detail & Related papers (2022-06-01T17:58:15Z) - Diffusion Causal Models for Counterfactual Estimation [18.438307666925425]
We consider the task of counterfactual estimation from observational imaging data given a known causal structure.
We propose Diff-SCM, a deep structural causal model that builds on recent advances of generative energy-based models.
We find that Diff-SCM produces more realistic and minimal counterfactuals than baselines on MNIST data and can also be applied to ImageNet data.
arXiv Detail & Related papers (2022-02-21T12:23:01Z) - Unsupervised Alternating Optimization for Blind Hyperspectral Imagery
Super-resolution [40.350308926790255]
This paper proposes an unsupervised blind HSI SR method to handle blind HSI fusion problem.
We first propose an alternating optimization based deep framework to estimate the degeneration models and reconstruct the latent image.
Then, a meta-learning based mechanism is further proposed to pre-train the network, which can effectively improve the speed and generalization ability.
arXiv Detail & Related papers (2020-12-03T07:52:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.