Deformable NeRF using Recursively Subdivided Tetrahedra

• URL: http://arxiv.org/abs/2410.04402v1
• Date: Sun, 6 Oct 2024 08:35:01 GMT
• Title: Deformable NeRF using Recursively Subdivided Tetrahedra
• Abstract summary: DeformRF is a method that seamlessly integrates the manipulability of tetrahedral meshes with the high-quality rendering capabilities of feature grid representations. We conduct a comprehensive evaluation of our DeformRF on both synthetic and real-captured datasets.
• Score: 24.080838312186376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While neural radiance fields (NeRF) have shown promise in novel view synthesis, their implicit representation limits explicit control over object manipulation. Existing research has proposed the integration of explicit geometric proxies to enable deformation. However, these methods face two primary challenges: firstly, the time-consuming and computationally demanding tetrahedralization process; and secondly, handling complex or thin structures often leads to either excessive, storage-intensive tetrahedral meshes or poor-quality ones that impair deformation capabilities. To address these challenges, we propose DeformRF, a method that seamlessly integrates the manipulability of tetrahedral meshes with the high-quality rendering capabilities of feature grid representations. To avoid ill-shaped tetrahedra and tetrahedralization for each object, we propose a two-stage training strategy. Starting with an almost-regular tetrahedral grid, our model initially retains key tetrahedra surrounding the object and subsequently refines object details using finer-granularity mesh in the second stage. We also present the concept of recursively subdivided tetrahedra to create higher-resolution meshes implicitly. This enables multi-resolution encoding while only necessitating the storage of the coarse tetrahedral mesh generated in the first training stage. We conduct a comprehensive evaluation of our DeformRF on both synthetic and real-captured datasets. Both quantitative and qualitative results demonstrate the effectiveness of our method for novel view synthesis and deformation tasks. Project page: https://ustc3dv.github.io/DeformRF/

Related papers

• HoloTetSphere: Unified TetSphere Mesh Reconstruction for Physical Simulations [16.69225871436957]
  We propose a topology-adaptive framework for holistic tetrahedral mesh reconstruction through end-to-end topological and geometric optimization.<n>Our method outperforms state-of-the-art techniques by achieving superior geometric accuracy and producing coherent, single-connected tetrahedral meshes.
  arXiv  Detail & Related papers  (2026-07-09T12:21:43Z)
• Radiance Meshes for Volumetric Reconstruction [56.51690637804858]
  We introduce radiance meshes, a technique for representing radiance fields with constant density tetrahedral cells.<n>Our model is able to perform exact and fast volume rendering using both synthesisization and ray-tracing.<n>Our rendering method exactly evaluates the volume equation and enables high quality, real-time view on standard consumer hardware.
  arXiv  Detail & Related papers  (2025-12-03T18:57:03Z)
• Mesh Denoising Transformer [104.5404564075393]
  Mesh denoising is aimed at removing noise from input meshes while preserving their feature structures. SurfaceFormer is a pioneering Transformer-based mesh denoising framework. New representation known as Local Surface Descriptor captures local geometric intricacies. Denoising Transformer module receives the multimodal information and achieves efficient global feature aggregation.
  arXiv  Detail & Related papers  (2024-05-10T15:27:43Z)
• FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis [48.9652334528436]
  We introduce an innovative, training-free approach FouriScale from the perspective of frequency domain analysis. We replace the original convolutional layers in pre-trained diffusion models by incorporating a dilation technique along with a low-pass operation. Our method successfully balances the structural integrity and fidelity of generated images, achieving an astonishing capacity of arbitrary-size, high-resolution, and high-quality generation.
  arXiv  Detail & Related papers  (2024-03-19T17:59:33Z)
• Learning Dynamic Tetrahedra for High-Quality Talking Head Synthesis [31.90503003079933]
  We introduce Dynamic Tetrahedra (DynTet), a novel hybrid representation that encodes explicit dynamic meshes by neural networks. Compared with prior works, DynTet demonstrates significant improvements in fidelity, lip synchronization, and real-time performance according to various metrics.
  arXiv  Detail & Related papers  (2024-02-27T09:56:15Z)
• Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking [52.393359791978035]
  Motion2VecSets is a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. We parameterize 4D dynamics with latent sets instead of using global latent codes. For more temporally-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames.
  arXiv  Detail & Related papers  (2024-01-12T15:05:08Z)
• TetGAN: A Convolutional Neural Network for Tetrahedral Mesh Generation [22.302674091474838]
  We present TetGAN, a convolutional neural network designed to generate tetrahedral meshes. Our formulation enables defining tetrahedral convolution, pooling, and upsampling operations. We illustrate the capabilities of our technique to encode tetrahedral meshes into a semantically meaningful latent-space.
  arXiv  Detail & Related papers  (2022-10-11T19:03:22Z)
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis [90.26556260531707]
  DMTet is a conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology.
  arXiv  Detail & Related papers  (2021-11-08T05:29:35Z)
• Recurrent Multi-view Alignment Network for Unsupervised Surface Registration [79.72086524370819]
  Learning non-rigid registration in an end-to-end manner is challenging due to the inherent high degrees of freedom and the lack of labeled training data. We propose to represent the non-rigid transformation with a point-wise combination of several rigid transformations. We also introduce a differentiable loss function that measures the 3D shape similarity on the projected multi-view 2D depth images.
  arXiv  Detail & Related papers  (2020-11-24T14:22:42Z)
• Learning Deformable Tetrahedral Meshes for 3D Reconstruction [78.0514377738632]
  3D shape representations that accommodate learning-based 3D reconstruction are an open problem in machine learning and computer graphics. Previous work on neural 3D reconstruction demonstrated benefits, but also limitations, of point cloud, voxel, surface mesh, and implicit function representations. We introduce Deformable Tetrahedral Meshes (DefTet) as a particular parameterization that utilizes volumetric tetrahedral meshes for the reconstruction problem.
  arXiv  Detail & Related papers  (2020-11-03T02:57:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.