Deep Spectral Meshes: Multi-Frequency Facial Mesh Processing with Graph Neural Networks

• URL: http://arxiv.org/abs/2402.10365v1
• Date: Thu, 15 Feb 2024 23:17:08 GMT
• Title: Deep Spectral Meshes: Multi-Frequency Facial Mesh Processing with Graph Neural Networks
• Authors: Robert Kosk, Richard Southern, Lihua You, Shaojun Bian, Willem Kokke, Greg Maguire
• Abstract summary: spectral meshes are introduced as a method to decompose mesh deformations into low- and high-frequency deformations. A parametric model for 3D facial mesh synthesis is built upon the proposed framework. Our model takes further advantage of spectral partitioning by representing different frequency levels with disparate, more suitable representations.
• Score: 1.170907599257096
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the rising popularity of virtual worlds, the importance of data-driven parametric models of 3D meshes has grown rapidly. Numerous applications, such as computer vision, procedural generation, and mesh editing, vastly rely on these models. However, current approaches do not allow for independent editing of deformations at different frequency levels. They also do not benefit from representing deformations at different frequencies with dedicated representations, which would better expose their properties and improve the generated meshes' geometric and perceptual quality. In this work, spectral meshes are introduced as a method to decompose mesh deformations into low-frequency and high-frequency deformations. These features of low- and high-frequency deformations are used for representation learning with graph convolutional networks. A parametric model for 3D facial mesh synthesis is built upon the proposed framework, exposing user parameters that control disentangled high- and low-frequency deformations. Independent control of deformations at different frequencies and generation of plausible synthetic examples are mutually exclusive objectives. A Conditioning Factor is introduced to leverage these objectives. Our model takes further advantage of spectral partitioning by representing different frequency levels with disparate, more suitable representations. Low frequencies are represented with standardised Euclidean coordinates, and high frequencies with a normalised deformation representation (DR). This paper investigates applications of our proposed approach in mesh reconstruction, mesh interpolation, and multi-frequency editing. It is demonstrated that our method improves the overall quality of generated meshes on most datasets when considering both the $L_1$ norm and perceptual Dihedral Angle Mesh Error (DAME) metrics.

Related papers

• Point Cloud Denoising With Fine-Granularity Dynamic Graph Convolutional Networks [58.050130177241186]
  Noise perturbations often corrupt 3-D point clouds, hindering downstream tasks such as surface reconstruction, rendering, and further processing. This paper introduces finegranularity dynamic graph convolutional networks called GDGCN, a novel approach to denoising in 3-D point clouds.
  arXiv  Detail & Related papers  (2024-11-21T14:19:32Z)
• 3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction [50.07071392673984]
  Existing methods learn 3D rotations parametrized in the spatial domain using angles or quaternions. We propose a frequency-domain approach that directly predicts Wigner-D coefficients for 3D rotation regression. Our method achieves state-of-the-art results on benchmarks such as ModelNet10-SO(3) and PASCAL3D+.
  arXiv  Detail & Related papers  (2024-11-01T12:50:38Z)
• Ultron: Enabling Temporal Geometry Compression of 3D Mesh Sequences using Temporal Correspondence and Mesh Deformation [2.0914328542137346]
  Existing 3D model compression methods primarily focus on static models and do not consider inter-frame information. This paper proposes a method to compress mesh sequences with arbitrary topology using temporal correspondence and mesh deformation.
  arXiv  Detail & Related papers  (2024-09-08T16:34:19Z)
• Mesh-based Gaussian Splatting for Real-time Large-scale Deformation [58.18290393082119]
  It is challenging for users to directly deform or manipulate implicit representations with large deformations in the real-time fashion. We develop a novel GS-based method that enables interactive deformation. Our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate.
  arXiv  Detail & Related papers  (2024-02-07T12:36:54Z)
• SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization [6.886220026399106]
  We introduce Multi-View Stereo (SD-MVS) to tackle challenges in 3D reconstruction of textureless areas. We are the first to adopt the Segment Anything Model (SAM) to distinguish semantic instances in scenes. We propose a unique refinement strategy that combines spherical coordinates and gradient descent on normals and pixelwise search interval on depths.
  arXiv  Detail & Related papers  (2024-01-12T05:25:57Z)
• Learning Modulated Transformation in GANs [69.95217723100413]
  We equip the generator in generative adversarial networks (GANs) with a plug-and-play module, termed as modulated transformation module (MTM) MTM predicts spatial offsets under the control of latent codes, based on which the convolution operation can be applied at variable locations. It is noteworthy that towards human generation on the challenging TaiChi dataset, we improve the FID of StyleGAN3 from 21.36 to 13.60, demonstrating the efficacy of learning modulated geometry transformation.
  arXiv  Detail & Related papers  (2023-08-29T17:51:22Z)
• High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition [77.29516516532439]
  We design a frequency split network to generate 3D hand mesh using different frequency bands in a coarse-to-fine manner. To capture high-frequency personalized details, we transform the 3D mesh into the frequency domain, and propose a novel frequency decomposition loss. Our approach generates fine-grained details for high-fidelity 3D hand reconstruction.
  arXiv  Detail & Related papers  (2023-07-08T19:26:09Z)
• Neural Volumetric Mesh Generator [40.224769507878904]
  We propose Neural Volumetric Mesh Generator(NVMG) which can generate novel and high-quality volumetric meshes. Our pipeline can generate high-quality artifact-free volumetric and surface meshes from random noise or a reference image without any post-processing.
  arXiv  Detail & Related papers  (2022-10-06T18:46:51Z)
• Simpler is better: spectral regularization and up-sampling techniques for variational autoencoders [1.2234742322758418]
  characterization of the spectral behavior of generative models based on neural networks remains an open issue. Recent research has focused heavily on generative adversarial networks and the high-frequency discrepancies between real and generated images. We propose a simple 2D Fourier transform-based spectral regularization loss for the Variational Autoencoders (VAEs)
  arXiv  Detail & Related papers  (2022-01-19T11:49:57Z)
• Dense Non-Rigid Structure from Motion: A Manifold Viewpoint [162.88686222340962]
  Non-Rigid Structure-from-Motion (NRSfM) problem aims to recover 3D geometry of a deforming object from its 2D feature correspondences across multiple frames. We show that our approach significantly improves accuracy, scalability, and robustness against noise.
  arXiv  Detail & Related papers  (2020-06-15T09:15:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.