Learning Spatially Collaged Fourier Bases for Implicit Neural Representation

• URL: http://arxiv.org/abs/2312.17018v1
• Date: Thu, 28 Dec 2023 13:36:23 GMT
• Title: Learning Spatially Collaged Fourier Bases for Implicit Neural Representation
• Authors: Jason Chun Lok Li, Chang Liu, Binxiao Huang and Ngai Wong
• Abstract summary: We introduce a learnable spatial mask that dispatches distinct Fourier bases into respective regions. This translates into collaging Fourier patches, thus enabling an accurate representation of complex signals. Our method outperforms all other baselines, improving the image fitting PSNR by over 3dB and 3D reconstruction to 98.81 IoU and 0.0011 Chamfer Distance.
• Score: 8.908709108907175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing approaches to Implicit Neural Representation (INR) can be interpreted as a global scene representation via a linear combination of Fourier bases of different frequencies. However, such universal basis functions can limit the representation capability in local regions where a specific component is unnecessary, resulting in unpleasant artifacts. To this end, we introduce a learnable spatial mask that effectively dispatches distinct Fourier bases into respective regions. This translates into collaging Fourier patches, thus enabling an accurate representation of complex signals. Comprehensive experiments demonstrate the superior reconstruction quality of the proposed approach over existing baselines across various INR tasks, including image fitting, video representation, and 3D shape representation. Our method outperforms all other baselines, improving the image fitting PSNR by over 3dB and 3D reconstruction to 98.81 IoU and 0.0011 Chamfer Distance.

Related papers

• Implicit Neural Representations with Fourier Kolmogorov-Arnold Networks [4.499833362998488]
  Implicit neural representations (INRs) use neural networks to provide continuous and resolution-independent representations of complex signals. The proposed FKAN utilizes learnable activation functions modeled as Fourier series in the first layer to effectively control and learn the task-specific frequency components. Experimental results show that our proposed FKAN model outperforms three state-of-the-art baseline schemes.
  arXiv  Detail & Related papers  (2024-09-14T05:53:33Z)
• MuRF: Multi-Baseline Radiance Fields [117.55811938988256]
  We present Multi-Baseline Radiance Fields (MuRF), a feed-forward approach to solving sparse view synthesis. MuRF achieves state-of-the-art performance across multiple different baseline settings. We also show promising zero-shot generalization abilities on the Mip-NeRF 360 dataset.
  arXiv  Detail & Related papers  (2023-12-07T18:59:56Z)
• Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
  Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information. We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering. Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
  arXiv  Detail & Related papers  (2023-04-20T04:05:22Z)
• Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedra [31.654710376807593]
  This paper proposes to use an adaptive representation based on tetrahedra obtained by Delaunay instead of uniform subdivision or point-based representations. We show that such a representation enables efficient training and leads to state-of-the-art results.
  arXiv  Detail & Related papers  (2023-04-19T21:44:27Z)
• Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field [55.431697263581626]
  We propose a Compositional Neural Radiance Field (CNeRF) for semantic 3D-aware portrait synthesis and manipulation. CNeRF divides the image by semantic regions and learns an independent neural radiance field for each region, and finally fuses them and renders the complete image. Compared to state-of-the-art 3D-aware GAN methods, our approach enables fine-grained semantic region manipulation, while maintaining high-quality 3D-consistent synthesis.
  arXiv  Detail & Related papers  (2023-02-03T07:17:46Z)
• Factor Fields: A Unified Framework for Neural Fields and Beyond [50.29013417187368]
  We present Factor Fields, a novel framework for modeling and representing signals. Our framework accommodates several recent signal representations including NeRF, Plenoxels, EG3D, Instant-NGP, and TensoRF. Our representation achieves better image approximation quality on 2D image regression tasks, higher geometric quality when reconstructing 3D signed distance fields, and higher compactness for radiance field reconstruction tasks.
  arXiv  Detail & Related papers  (2023-02-02T17:06:50Z)
• Deep Fourier Up-Sampling [100.59885545206744]
  Up-sampling in the Fourier domain is more challenging as it does not follow such a local property. We propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues.
  arXiv  Detail & Related papers  (2022-10-11T06:17:31Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Learning Local Implicit Fourier Representation for Image Warping [11.526109213908091]
  We propose a local texture estimator for image warping (LTEW) followed by an implicit neural representation to deform images into continuous shapes. Our LTEW-based neural function outperforms existing warping methods for asymmetric-scale SR and homography transform.
  arXiv  Detail & Related papers  (2022-07-05T06:30:17Z)
• Seeing Implicit Neural Representations as Fourier Series [13.216389226310987]
  Implicit Neural Representations (INR) use multilayer perceptrons to represent high-frequency functions in low-dimensional problem domains. These representations achieved state-of-the-art results on tasks related to complex 3D objects and scenes. This work analyzes the connection between the two methods and shows that a Fourier mapped perceptron is structurally like one hidden layer SIREN.
  arXiv  Detail & Related papers  (2021-09-01T08:40:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.