Sharpening Neural Implicit Functions with Frequency Consolidation Priors

• URL: http://arxiv.org/abs/2412.19720v1
• Date: Fri, 27 Dec 2024 16:18:46 GMT
• Title: Sharpening Neural Implicit Functions with Frequency Consolidation Priors
• Authors: Chao Chen, Yu-Shen Liu, Zhizhong Han,
• Abstract summary: Signed Distance Functions (SDFs) are vital implicit representations to represent high fidelity 3D surfaces. Current methods mainly leverage a neural network to learn an SDF from various supervisions including signed, 3D point clouds, or multi-view images. We introduce a method to sharpen a low frequency SDF observation by recovering its high frequency components, pursuing a sharper and more complete surface.
• Score: 53.6277160912059
• License:
• Abstract: Signed Distance Functions (SDFs) are vital implicit representations to represent high fidelity 3D surfaces. Current methods mainly leverage a neural network to learn an SDF from various supervisions including signed distances, 3D point clouds, or multi-view images. However, due to various reasons including the bias of neural network on low frequency content, 3D unaware sampling, sparsity in point clouds, or low resolutions of images, neural implicit representations still struggle to represent geometries with high frequency components like sharp structures, especially for the ones learned from images or point clouds. To overcome this challenge, we introduce a method to sharpen a low frequency SDF observation by recovering its high frequency components, pursuing a sharper and more complete surface. Our key idea is to learn a mapping from a low frequency observation to a full frequency coverage in a data-driven manner, leading to a prior knowledge of shape consolidation in the frequency domain, dubbed frequency consolidation priors. To better generalize a learned prior to unseen shapes, we introduce to represent frequency components as embeddings and disentangle the embedding of the low frequency component from the embedding of the full frequency component. This disentanglement allows the prior to generalize on an unseen low frequency observation by simply recovering its full frequency embedding through a test-time self-reconstruction. Our evaluations under widely used benchmarks or real scenes show that our method can recover high frequency component and produce more accurate surfaces than the latest methods. The code, data, and pre-trained models are available at \url{https://github.com/chenchao15/FCP}.

Related papers

• Frequency-regularized Neural Representation Method for Sparse-view Tomographic Reconstruction [8.45338755060592]
  We introduce the Regularized Neural Attenuation/Activity Field (Freq-NAF) for self-supervised sparse-view tomographic reconstruction. Freq-NAF mitigates overfitting by frequency regularization, directly controlling the visible frequency bands in the neural network input.
  arXiv  Detail & Related papers  (2024-09-22T11:19:38Z)
• Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning [81.98675881423131]
  This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
  arXiv  Detail & Related papers  (2024-03-12T01:28:00Z)
• Frequency-Adaptive Pan-Sharpening with Mixture of Experts [22.28680499480492]
  We propose a novel Frequency Adaptive Mixture of Experts (FAME) learning framework for pan-sharpening. Our method performs the best against other state-of-the-art ones and comprises a strong generalization ability for real-world scenes.
  arXiv  Detail & Related papers  (2024-01-04T08:58:25Z)
• 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands [51.305421495638434]
  Neural radiance fields (NeRFs) are promising 3D representations for scenes, objects, and humans. This paper proposes a generalizable visibility-aware NeRF framework for interacting hands. Experiments on the Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms conventional NeRFs significantly.
  arXiv  Detail & Related papers  (2024-01-02T00:42:06Z)
• WaveNeRF: Wavelet-based Generalizable Neural Radiance Fields [149.2296890464997]
  We design WaveNeRF, which integrates wavelet frequency decomposition into MVS and NeRF. WaveNeRF achieves superior generalizable radiance field modeling when only given three images as input.
  arXiv  Detail & Related papers  (2023-08-09T09:24:56Z)
• Both Spatial and Frequency Cues Contribute to High-Fidelity Image Inpainting [9.080472817672263]
  Deep generative approaches have obtained great success in image inpainting recently. Most generative inpainting networks suffer from either over-smooth results or aliasing artifacts. We propose an effective Frequency-Spatial Complementary Network (FSCN) by exploiting rich semantic information in both spatial and frequency domains.
  arXiv  Detail & Related papers  (2023-07-15T01:52:06Z)
• High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition [77.29516516532439]
  We design a frequency split network to generate 3D hand mesh using different frequency bands in a coarse-to-fine manner. To capture high-frequency personalized details, we transform the 3D mesh into the frequency domain, and propose a novel frequency decomposition loss. Our approach generates fine-grained details for high-fidelity 3D hand reconstruction.
  arXiv  Detail & Related papers  (2023-07-08T19:26:09Z)
• Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in VIS and NIR Scenario [87.72258480670627]
  Existing face forgery detection methods based on frequency domain find that the GAN forged images have obvious grid-like visual artifacts in the frequency spectrum compared to the real images. This paper proposes a Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT) to achieve a more comprehensive spatial-temporal feature representation.
  arXiv  Detail & Related papers  (2022-07-05T09:27:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.