In Search of a Data Transformation That Accelerates Neural Field Training

• URL: http://arxiv.org/abs/2311.17094v2
• Date: Tue, 26 Mar 2024 13:21:43 GMT
• Title: In Search of a Data Transformation That Accelerates Neural Field Training
• Authors: Junwon Seo, Sangyoon Lee, Kwang In Kim, Jaeho Lee,
• Abstract summary: We focus on how permuting pixel locations affect the convergence speed of SGD. Counterly, we find that randomly permuting the pixel locations can considerably accelerate the training. Our analyses suggest that the random pixel permutations remove the easy-to-fit patterns, which hinder easy optimization in the early stage but capture fine details of the signal.
• Score: 37.39915075581319
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural field is an emerging paradigm in data representation that trains a neural network to approximate the given signal. A key obstacle that prevents its widespread adoption is the encoding speed-generating neural fields requires an overfitting of a neural network, which can take a significant number of SGD steps to reach the desired fidelity level. In this paper, we delve into the impacts of data transformations on the speed of neural field training, specifically focusing on how permuting pixel locations affect the convergence speed of SGD. Counterintuitively, we find that randomly permuting the pixel locations can considerably accelerate the training. To explain this phenomenon, we examine the neural field training through the lens of PSNR curves, loss landscapes, and error patterns. Our analyses suggest that the random pixel permutations remove the easy-to-fit patterns, which facilitate easy optimization in the early stage but hinder capturing fine details of the signal.

Related papers

• Accelerating Scalable Graph Neural Network Inference with Node-Adaptive Propagation [80.227864832092]
  Graph neural networks (GNNs) have exhibited exceptional efficacy in a diverse array of applications. The sheer size of large-scale graphs presents a significant challenge to real-time inference with GNNs. We propose an online propagation framework and two novel node-adaptive propagation methods.
  arXiv  Detail & Related papers  (2023-10-17T05:03:00Z)
• Training Robust Spiking Neural Networks with ViewPoint Transform and SpatioTemporal Stretching [4.736525128377909]
  We propose a novel data augmentation method, ViewPoint Transform and Spatio Stretching (VPT-STS) It improves the robustness of spiking neural networks by transforming the rotation centers and angles in thetemporal domain to generate samples from different viewpoints. Experiments on prevailing neuromorphic datasets demonstrate that VPT-STS is broadly effective on multi-event representations and significantly outperforms pure spatial geometric transformations.
  arXiv  Detail & Related papers  (2023-03-14T03:09:56Z)
• Random Weight Factorization Improves the Training of Continuous Neural Representations [1.911678487931003]
  Continuous neural representations have emerged as a powerful and flexible alternative to classical discretized representations of signals. We propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers. We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate.
  arXiv  Detail & Related papers  (2022-10-03T23:48:48Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Meta-Learning Sparse Implicit Neural Representations [69.15490627853629]
  Implicit neural representations are a promising new avenue of representing general signals. Current approach is difficult to scale for a large number of signals or a data set. We show that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models.
  arXiv  Detail & Related papers  (2021-10-27T18:02:53Z)
• SignalNet: A Low Resolution Sinusoid Decomposition and Estimation Network [79.04274563889548]
  We propose SignalNet, a neural network architecture that detects the number of sinusoids and estimates their parameters from quantized in-phase and quadrature samples. We introduce a worst-case learning threshold for comparing the results of our network relative to the underlying data distributions. In simulation, we find that our algorithm is always able to surpass the threshold for three-bit data but often cannot exceed the threshold for one-bit data.
  arXiv  Detail & Related papers  (2021-06-10T04:21:20Z)
• Adaptive Latent Space Tuning for Non-Stationary Distributions [62.997667081978825]
  We present a method for adaptive tuning of the low-dimensional latent space of deep encoder-decoder style CNNs. We demonstrate our approach for predicting the properties of a time-varying charged particle beam in a particle accelerator.
  arXiv  Detail & Related papers  (2021-05-08T03:50:45Z)
• A Light-Weighted Convolutional Neural Network for Bitemporal SAR Image Change Detection [40.58864817923371]
  We propose a lightweight neural network to reduce the computational and spatial complexity. In the proposed network, we replace normal convolutional layers with bottleneck layers that keep the same number of channels between input and output. We verify our light-weighted neural network on four sets of bitemporal SAR images.
  arXiv  Detail & Related papers  (2020-05-29T04:01:32Z)
• Investigating Generalization in Neural Networks under Optimally Evolved Training Perturbations [46.8676764079206]
  We study the generalization properties of neural networks under input perturbations. We show that minimal training data corruption by a few pixel modifications can cause drastic overfitting. We propose an evolutionary algorithm to search for optimal pixel perturbations.
  arXiv  Detail & Related papers  (2020-03-14T14:38:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.