Coordinate-Aware Modulation for Neural Fields

• URL: http://arxiv.org/abs/2311.14993v1
• Date: Sat, 25 Nov 2023 10:42:51 GMT
• Title: Coordinate-Aware Modulation for Neural Fields
• Authors: Joo Chan Lee, Daniel Rho, Seungtae Nam, Jong Hwan Ko, Eunbyung Park
• Abstract summary: We propose a novel way for exploiting both synthesiss and grid representations in neural fields. We suggest a Neural Coordinate-Aware Modulation (CAM), which modulates the parameters using scale and shift features extracted from the grid representations.
• Score: 11.844561374381575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural fields, mapping low-dimensional input coordinates to corresponding signals, have shown promising results in representing various signals. Numerous methodologies have been proposed, and techniques employing MLPs and grid representations have achieved substantial success. MLPs allow compact and high expressibility, yet often suffer from spectral bias and slow convergence speed. On the other hand, methods using grids are free from spectral bias and achieve fast training speed, however, at the expense of high spatial complexity. In this work, we propose a novel way for exploiting both MLPs and grid representations in neural fields. Unlike the prevalent methods that combine them sequentially (extract features from the grids first and feed them to the MLP), we inject spectral bias-free grid representations into the intermediate features in the MLP. More specifically, we suggest a Coordinate-Aware Modulation (CAM), which modulates the intermediate features using scale and shift parameters extracted from the grid representations. This can maintain the strengths of MLPs while mitigating any remaining potential biases, facilitating the rapid learning of high-frequency components. In addition, we empirically found that the feature normalizations, which have not been successful in neural filed literature, proved to be effective when applied in conjunction with the proposed CAM. Experimental results demonstrate that CAM enhances the performance of neural representation and improves learning stability across a range of signals. Especially in the novel view synthesis task, we achieved state-of-the-art performance with the least number of parameters and fast training speed for dynamic scenes and the best performance under 1MB memory for static scenes. CAM also outperforms the best-performing video compression methods using neural fields by a large margin.

Related papers

• FreSh: Frequency Shifting for Accelerated Neural Representation Learning [11.175745750843484]
  Implicit Neural Representations (INRs) have recently gained attention as a powerful approach for continuously representing signals such as images, videos, and 3D shapes using multilayer perceptrons (MLPs) Low-frequency details are known to exhibit a low-frequency bias, limiting their ability to capture high-frequency details accurately. We propose frequency shifting (or FreSh) to align the frequency spectrum of the initial output with that of the target signal.
  arXiv  Detail & Related papers  (2024-10-07T14:05:57Z)
• Attention Beats Linear for Fast Implicit Neural Representation Generation [13.203243059083533]
  We propose Attention-based Localized INR (ANR) composed of a localized attention layer (LAL) and a global representation vector. With instance-specific representation and instance-agnostic ANR parameters, the target signals are well reconstructed as a continuous function.
  arXiv  Detail & Related papers  (2024-07-22T03:52:18Z)
• KFD-NeRF: Rethinking Dynamic NeRF with Kalman Filter [49.85369344101118]
  We introduce KFD-NeRF, a novel dynamic neural radiance field integrated with an efficient and high-quality motion reconstruction framework based on Kalman filtering. Our key idea is to model the dynamic radiance field as a dynamic system whose temporally varying states are estimated based on two sources of knowledge: observations and predictions. Our KFD-NeRF demonstrates similar or even superior performance within comparable computational time and state-of-the-art view synthesis performance with thorough training.
  arXiv  Detail & Related papers  (2024-07-18T05:48:24Z)
• Synergistic Integration of Coordinate Network and Tensorial Feature for Improving Neural Radiance Fields from Sparse Inputs [26.901819636977912]
  We propose a method that integrates multi-plane representation with a coordinate-based network known for strong bias toward low-frequency signals. We demonstrate that our proposed method outperforms baseline models for both static and dynamic NeRFs with sparse inputs.
  arXiv  Detail & Related papers  (2024-05-13T15:42:46Z)
• On Optimal Sampling for Learning SDF Using MLPs Equipped with Positional Encoding [79.67071790034609]
  We devise a tool to determine the appropriate sampling rate for learning an accurate neural implicit field without undesirable side effects. It is observed that a PE-equipped has an intrinsic frequency much higher than the highest frequency component in the PE layer. We empirically show in the setting of SDF fitting, this recommended sampling rate is sufficient to secure accurate fitting results.
  arXiv  Detail & Related papers  (2024-01-02T10:51:52Z)
• DNS SLAM: Dense Neural Semantic-Informed SLAM [92.39687553022605]
  DNS SLAM is a novel neural RGB-D semantic SLAM approach featuring a hybrid representation. Our method integrates multi-view geometry constraints with image-based feature extraction to improve appearance details. Our experimental results achieve state-of-the-art performance on both synthetic data and real-world data tracking.
  arXiv  Detail & Related papers  (2023-11-30T21:34:44Z)
• Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering [3.8200916793910973]
  Recent advances in Neural Radiance Fields (NeRF) have demonstrated significant potential for representing 3D scene appearances as implicit neural networks. However, the lengthy training and rendering process hinders the widespread adoption of this promising technique for real-time rendering applications. We present an effective adaptive multi-NeRF method designed to accelerate the neural rendering process for large scenes.
  arXiv  Detail & Related papers  (2023-10-03T08:34:49Z)
• ResFields: Residual Neural Fields for Spatiotemporal Signals [61.44420761752655]
  ResFields is a novel class of networks specifically designed to effectively represent complex temporal signals. We conduct comprehensive analysis of the properties of ResFields and propose a matrix factorization technique to reduce the number of trainable parameters. We demonstrate the practical utility of ResFields by showcasing its effectiveness in capturing dynamic 3D scenes from sparse RGBD cameras.
  arXiv  Detail & Related papers  (2023-09-06T16:59:36Z)
• Meta-Learning Sparse Implicit Neural Representations [69.15490627853629]
  Implicit neural representations are a promising new avenue of representing general signals. Current approach is difficult to scale for a large number of signals or a data set. We show that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models.
  arXiv  Detail & Related papers  (2021-10-27T18:02:53Z)
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains [69.62456877209304]
  We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron to learn high-frequency functions. Results shed light on advances in computer vision and graphics that achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2020-06-18T17:59:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.