Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC

• URL: http://arxiv.org/abs/2312.05705v4
• Date: Tue, 23 Jul 2024 12:13:44 GMT
• Title: Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC
• Authors: Wu Lin, Felix Dangel, Runa Eschenhagen, Kirill Neklyudov, Agustinus Kristiadi, Richard E. Turner, Alireza Makhzani,
• Abstract summary: Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense. We formulate an inverse-free KFAC update and impose structures in the Kronecker factors, resulting in structured inverse-free gradient natural descent.
• Score: 26.275682325827706
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training.

Related papers

• Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization [52.16435732772263]
  Second-order optimization has been shown to accelerate the training of deep neural networks in many applications. However, generalization properties of second-order methods are still being debated. We show for the first time that exact Gauss-Newton (GN) updates take on a tractable form in a class of deep architectures.
  arXiv  Detail & Related papers  (2024-11-12T17:58:40Z)
• Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks [3.7308074617637588]
  We propose Kronecker-factored approximate curvature (KFAC) for PINN losses that greatly reduces the computational cost and allows scaling to much larger networks. We find that our KFAC-based gradients are competitive with expensive second-order methods on small problems, scale more favorably to higher-dimensional neural networks and PDEs, and consistently outperform first-order methods and LBFGS.
  arXiv  Detail & Related papers  (2024-05-24T14:36:02Z)
• Inverse-Free Fast Natural Gradient Descent Method for Deep Learning [52.0693420699086]
  We present a fast natural gradient descent (FNGD) method that only requires inversion during the first epoch. FNGD exhibits similarities to the average sum in first-order methods, leading to the computational complexity of FNGD being comparable to that of first-order methods.
  arXiv  Detail & Related papers  (2024-03-06T05:13:28Z)
• Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures [85.76673783330334]
  Two different settings of linear weight-sharing layers motivate two flavours of Kronecker-Factored Approximate Curvature (K-FAC) We show they are exact for deep linear networks with weight-sharing in their respective setting. We observe little difference between these two K-FAC variations when using them to train both a graph neural network and a vision transformer.
  arXiv  Detail & Related papers  (2023-11-01T16:37:00Z)
• Guaranteed Approximation Bounds for Mixed-Precision Neural Operators [83.64404557466528]
  We build on intuition that neural operator learning inherently induces an approximation error. We show that our approach reduces GPU memory usage by up to 50% and improves throughput by 58% with little or no reduction in accuracy.
  arXiv  Detail & Related papers  (2023-07-27T17:42:06Z)
• Brand New K-FACs: Speeding up K-FAC with Online Decomposition Updates [0.0]
  We exploit the exponential-average construction paradigm of the K-factors, and use online numerical linear algebra techniques. We propose a K-factor inverse update which scales linearly in layer size. We also propose an inverse application procedure which scales linearly as well.
  arXiv  Detail & Related papers  (2022-10-16T09:41:23Z)
• Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization [0.913755431537592]
  We show that Kronecker-Factored, block-diagonal curvature estimates (KFAC) significantly outperforms true second-order updates. We also show that KFAC approximates a first-order gradient algorithm, which performs a gradient descent on rather than weights.
  arXiv  Detail & Related papers  (2022-01-28T17:06:26Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• Kronecker-factored Quasi-Newton Methods for Convolutional Neural Networks [10.175972095073282]
  KF-QN-CNN is a new quasi-factored training convolutional neural networks (CNNs) KF-QN-CNN consistently exhibited superior performance in all of our tests.
  arXiv  Detail & Related papers  (2021-02-12T19:40:34Z)
• An iterative K-FAC algorithm for Deep Learning [0.0]
  Key of K-FAC is to approximates Fisher information matrix (FIM) as a block-diagonal matrix where each block is an inverse of tiny Kronecker factors. In this short note, we present CG-FAC -- an new iterative K-FAC algorithm. We prove that the time and memory complexity of iterative CG-FAC is much less than that of standard K-FAC algorithm.
  arXiv  Detail & Related papers  (2021-01-01T12:04:01Z)
• Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification [53.50708351813565]
  We propose SVD training, the first method to explicitly achieve low-rank DNNs during training without applying SVD on every step. We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy.
  arXiv  Detail & Related papers  (2020-04-20T02:40:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.