Flexible and Efficient Surrogate Gradient Modeling with Forward Gradient Injection

• URL: http://arxiv.org/abs/2406.00177v1
• Date: Fri, 31 May 2024 20:14:35 GMT
• Title: Flexible and Efficient Surrogate Gradient Modeling with Forward Gradient Injection
• Abstract summary: PyTorch, for example, allows the custom specification of the backward pass of an operation by overriding its backward method. While these methods are common practice and usually work well, they also have several disadvantages such as limited flexibility, additional source code overhead, poor usability, or a potentially strong negative impact on the effectiveness of automatic model optimization procedures. In this paper, an alternative way to formulate surrogate gradients is presented, namely, forward gradient injection (FGI)
• Score: 2.0577627277681887
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic differentiation is a key feature of present deep learning frameworks. Moreover, they typically provide various ways to specify custom gradients within the computation graph, which is of particular importance for defining surrogate gradients in the realms of non-differentiable operations such as the Heaviside function in spiking neural networks (SNNs). PyTorch, for example, allows the custom specification of the backward pass of an operation by overriding its backward method. Other frameworks provide comparable options. While these methods are common practice and usually work well, they also have several disadvantages such as limited flexibility, additional source code overhead, poor usability, or a potentially strong negative impact on the effectiveness of automatic model optimization procedures. In this paper, an alternative way to formulate surrogate gradients is presented, namely, forward gradient injection (FGI). FGI applies a simple but effective combination of basic standard operations to inject an arbitrary gradient shape into the computational graph directly within the forward pass. It is demonstrated that using FGI is straightforward and convenient. Moreover, it is shown that FGI can significantly increase the model performance in comparison to custom backward methods in SNNs when using TorchScript. These results are complemented with a general performance study on recurrent SNNs with TorchScript and torch.compile, revealing the potential for a training speedup of more than 7x and an inference speedup of more than 16x in comparison with pure PyTorch.

Related papers

• Gradient Descent with Provably Tuned Learning-rate Schedules [14.391648046717073]
  We develop novel analytical tools for provably tuning factors in gradient-based algorithms.<n>Our analysis applies to neural networks with commonly used activation functions.
  arXiv  Detail & Related papers  (2025-12-04T18:49:58Z)
• AutoGD: Automatic Learning Rate Selection for Gradient Descent [54.195493042469]
  We introduce AutoGD: a gradient descent method that automatically determines whether to increase or decrease the learning rate at a given iteration.<n>We show that we can recover the optimal rate of GD (up to a constant) for a broad class of functions without knowledge of smoothness constants.
  arXiv  Detail & Related papers  (2025-10-10T23:47:22Z)
• HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuning [20.308785668386424]
  We propose an approach that emulates gradient descent without computing gradients, enabling efficient test-time adaptation. Specifically, we formulate gradient descent as an Euler discretization of an ordinary differential equation (ODE) and train an auxiliary network to predict the task-conditional drift. The adaptation then reduces to a simple numerical integration, which requires only a few forward passes of the auxiliary network.
  arXiv  Detail & Related papers  (2025-04-21T03:04:38Z)
• Deep Implicit Optimization for Robust and Flexible Image Registration [20.34181966545357]
  We bridge the gap between classical and learning methods by incorporating optimization as a layer in a deep network. By implicitly differentiating end-to-end through an iterative optimization solver, our learned features are registration and label-aware. Our framework shows excellent performance on in-domain datasets, and is agnostic to domain shift.
  arXiv  Detail & Related papers  (2024-06-11T15:28:48Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Variational Bayes image restoration with compressive autoencoders [6.689746581015932]
  Regularization of inverse problems is paramount to importance in computational imaging.<n>In this work, we first propose to use variational autoencoders instead of state-of-the-art generative models.<n>As a second contribution, we introduce the Variational Bayes Latent Estimation (VBLE) algorithm, which performs latent estimation within variational inference.
  arXiv  Detail & Related papers  (2023-11-29T15:49:31Z)
• ELRA: Exponential learning rate adaption gradient descent optimization method [83.88591755871734]
  We present a novel, fast (exponential rate), ab initio (hyper-free) gradient based adaption. The main idea of the method is to adapt the $alpha by situational awareness. It can be applied to problems of any dimensions n and scales only linearly.
  arXiv  Detail & Related papers  (2023-09-12T14:36:13Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  We present Layer-wise Feedback Propagation (LFP), a novel training principle for neural network-like predictors. LFP decomposes a reward to individual neurons based on their respective contributions to solving a given task. Our method then implements a greedy approach reinforcing helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics [13.558059987555234]
  We show how to automatically optimize graphics for undefined or zero gradients. The fitting is performed online, alongside the parameter optimization, and self-supervised, without pre-computed data or pre-trained models. In contrast to other algorithms, our approach scales well to higher dimensions, which we demonstrate on problems with up to 35k variables.
  arXiv  Detail & Related papers  (2023-08-10T17:57:22Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Scaling Forward Gradient With Local Losses [117.22685584919756]
  Forward learning is a biologically plausible alternative to backprop for learning deep neural networks. We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
  arXiv  Detail & Related papers  (2022-10-07T03:52:27Z)
• Joint inference and input optimization in equilibrium networks [68.63726855991052]
  deep equilibrium model is a class of models that foregoes traditional network depth and instead computes the output of a network by finding the fixed point of a single nonlinear layer. We show that there is a natural synergy between these two settings. We demonstrate this strategy on various tasks such as training generative models while optimizing over latent codes, training models for inverse problems like denoising and inpainting, adversarial training and gradient based meta-learning.
  arXiv  Detail & Related papers  (2021-11-25T19:59:33Z)
• GuideBP: Guiding Backpropagation Through Weaker Pathways of Parallel Logits [6.764324841419295]
  The proposed approach guides the gradients of backpropagation along weakest concept representations. A weakness scores defines the class specific performance of individual pathways which is then used to create a logit. The proposed approach has been shown to perform better than traditional column merging techniques.
  arXiv  Detail & Related papers  (2021-04-23T14:14:00Z)
• Population Gradients improve performance across data-sets and architectures in object classification [6.17047113475566]
  We present a new method to calculate the gradients while training Neural Networks (NNs) It significantly improves final performance across architectures, data-sets, hyper- parameter values, training length, and model sizes. Besides being effective in the wide array situations that we have tested, the increase in performance (e.g. F1) is as high or higher than this one of all the other widespread performance-improving methods.
  arXiv  Detail & Related papers  (2020-10-23T09:40:23Z)
• Gradient Centralization: A New Optimization Technique for Deep Neural Networks [74.935141515523]
  gradient centralization (GC) operates directly on gradients by centralizing the gradient vectors to have zero mean. GC can be viewed as a projected gradient descent method with a constrained loss function. GC is very simple to implement and can be easily embedded into existing gradient based DNNs with only one line of code.
  arXiv  Detail & Related papers  (2020-04-03T10:25:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.