EVOS: Efficient Implicit Neural Training via EVOlutionary Selector

• URL: http://arxiv.org/abs/2412.10153v1
• Date: Fri, 13 Dec 2024 14:11:42 GMT
• Title: EVOS: Efficient Implicit Neural Training via EVOlutionary Selector
• Authors: Weixiang Zhang, Shuzhao Xie, Chengwei Ren, Siyi Xie, Chen Tang, Shijia Ge, Mingzi Wang, Zhi Wang,
• Abstract summary: EVOlutionary Selector (EVOS) is an efficient training paradigm for accelerating Implicit Neural Representation (INR) We treat each sample as an individual in an evolutionary process, where only those fittest ones survive and merit inclusion in training. Our method achieves approximately 48%-66% reduction in training time while ensuring superior convergence without additional cost.
• Score: 10.985825930928463
• License:
• Abstract: We propose EVOlutionary Selector (EVOS), an efficient training paradigm for accelerating Implicit Neural Representation (INR). Unlike conventional INR training that feeds all samples through the neural network in each iteration, our approach restricts training to strategically selected points, reducing computational overhead by eliminating redundant forward passes. Specifically, we treat each sample as an individual in an evolutionary process, where only those fittest ones survive and merit inclusion in training, adaptively evolving with the neural network dynamics. While this is conceptually similar to Evolutionary Algorithms, their distinct objectives (selection for acceleration vs. iterative solution optimization) require a fundamental redefinition of evolutionary mechanisms for our context. In response, we design sparse fitness evaluation, frequency-guided crossover, and augmented unbiased mutation to comprise EVOS. These components respectively guide sample selection with reduced computational cost, enhance performance through frequency-domain balance, and mitigate selection bias from cached evaluation. Extensive experiments demonstrate that our method achieves approximately 48%-66% reduction in training time while ensuring superior convergence without additional cost, establishing state-of-the-art acceleration among recent sampling-based strategies.

Related papers

• Preventing Local Pitfalls in Vector Quantization via Optimal Transport [77.15924044466976]
  We introduce OptVQ, a novel vector quantization method that employs the Sinkhorn algorithm to optimize the optimal transport problem. Our experiments on image reconstruction tasks demonstrate that OptVQ achieves 100% codebook utilization and surpasses current state-of-the-art VQNs in reconstruction quality.
  arXiv  Detail & Related papers  (2024-12-19T18:58:14Z)
• Adaptive Class Emergence Training: Enhancing Neural Network Stability and Generalization through Progressive Target Evolution [0.0]
  We propose a novel training methodology for neural networks in classification problems. We evolve the target outputs from a null vector to one-hot encoded vectors throughout the training process. This gradual transition allows the network to adapt more smoothly to the increasing complexity of the classification task.
  arXiv  Detail & Related papers  (2024-09-04T03:25:48Z)
• Switchable Decision: Dynamic Neural Generation Networks [98.61113699324429]
  We propose a switchable decision to accelerate inference by dynamically assigning resources for each data instance. Our method benefits from less cost during inference while keeping the same accuracy.
  arXiv  Detail & Related papers  (2024-05-07T17:44:54Z)
• Optimal feature rescaling in machine learning based on neural networks [0.0]
  An optimal rescaling of input features (OFR) is carried out by a Genetic Algorithm (GA) The OFR reshapes the input space improving the conditioning of the gradient-based algorithm used for the training. The approach has been tested on a FFNN modeling the outcome of a real industrial process.
  arXiv  Detail & Related papers  (2024-02-13T21:57:31Z)
• Lottery Tickets in Evolutionary Optimization: On Sparse Backpropagation-Free Trainability [0.0]
  We study gradient descent (GD)-based sparse training and evolution strategies (ES) We find that ES explore diverse and flat local optima and do not preserve linear mode connectivity across sparsity levels and independent runs.
  arXiv  Detail & Related papers  (2023-05-31T15:58:54Z)
• One Forward is Enough for Neural Network Training via Likelihood Ratio Method [47.013384887197454]
  Backpropagation (BP) is the mainstream approach for gradient computation in neural network training. We develop a unified likelihood ratio (ULR) method for estimation with just one forward propagation.
  arXiv  Detail & Related papers  (2023-05-15T19:02:46Z)
• Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies [50.10277748405355]
  Noise-Reuse Evolution Strategies (NRES) is a general class of unbiased online evolution strategies methods. We show NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of steps across a variety of applications.
  arXiv  Detail & Related papers  (2023-04-21T17:53:05Z)
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks [56.35403810762512]
  Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
  arXiv  Detail & Related papers  (2023-02-01T04:22:59Z)
• Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics [85.31710759801705]
  Current practice requires expensive computational costs in model training for performance prediction. We propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections.
  arXiv  Detail & Related papers  (2022-01-11T20:53:15Z)
• A Hybrid Method for Training Convolutional Neural Networks [3.172761915061083]
  We propose a hybrid method that uses both backpropagation and evolutionary strategies to train Convolutional Neural Networks. We show that the proposed hybrid method is capable of improving upon regular training in the task of image classification.
  arXiv  Detail & Related papers  (2020-04-15T17:52:48Z)
• Subset Sampling For Progressive Neural Network Learning [106.12874293597754]
  Progressive Neural Network Learning is a class of algorithms that incrementally construct the network's topology and optimize its parameters based on the training data. We propose to speed up this process by exploiting subsets of training data at each incremental training step. Experimental results in object, scene and face recognition problems demonstrate that the proposed approach speeds up the optimization procedure considerably.
  arXiv  Detail & Related papers  (2020-02-17T18:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.