Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

• URL: http://arxiv.org/abs/2409.04707v1
• Date: Sat, 7 Sep 2024 04:37:20 GMT
• Title: Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training
• Authors: Yuhan Ma, Dan Sun, Erdi Gao, Ningjing Sang, Iris Li, Guanming Huang,
• Abstract summary: This paper explores the relationship between optimization theory and deep learning. We introduce an enhancement to the descent algorithm, highlighting its variants, which are the cornerstone of neural networks. Our experiments on diverse deep learning tasks substantiate the improved algorithm's efficacy.
• Score: 0.036651088217486416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimization theory serves as a pivotal scientific instrument for achieving optimal system performance, with its origins in economic applications to identify the best investment strategies for maximizing benefits. Over the centuries, from the geometric inquiries of ancient Greece to the calculus contributions by Newton and Leibniz, optimization theory has significantly advanced. The persistent work of scientists like Lagrange, Cauchy, and von Neumann has fortified its progress. The modern era has seen an unprecedented expansion of optimization theory applications, particularly with the growth of computer science, enabling more sophisticated computational practices and widespread utilization across engineering, decision analysis, and operations research. This paper delves into the profound relationship between optimization theory and deep learning, highlighting the omnipresence of optimization problems in the latter. We explore the gradient descent algorithm and its variants, which are the cornerstone of optimizing neural networks. The chapter introduces an enhancement to the SGD optimizer, drawing inspiration from numerical optimization methods, aiming to enhance interpretability and accuracy. Our experiments on diverse deep learning tasks substantiate the improved algorithm's efficacy. The paper concludes by emphasizing the continuous development of optimization theory and its expanding role in solving intricate problems, enhancing computational capabilities, and informing better policy decisions.

Related papers

• PINN-BO: A Black-box Optimization Algorithm using Physics-Informed Neural Networks [11.618811218101058]
  Black-box optimization is a powerful approach for discovering global optima in noisy and expensive black-box functions. We propose PINN-BO, a black-box optimization algorithm employing Physics-Informed Neural Networks. We show that our algorithm is more sample-efficient compared to existing methods.
  arXiv  Detail & Related papers  (2024-02-05T17:58:17Z)
• An Invariant Information Geometric Method for High-Dimensional Online Optimization [9.538618632613714]
  We introduce a full invariance oriented evolution strategies algorithm, derived from its corresponding framework. We benchmark SynCMA against leading algorithms in Bayesian optimization and evolution strategies. In all scenarios, SynCMA demonstrates great competence, if not dominance, over other algorithms in sample efficiency.
  arXiv  Detail & Related papers  (2024-01-03T07:06:26Z)
• Analyzing and Enhancing the Backward-Pass Convergence of Unrolled Optimization [50.38518771642365]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. A central challenge in this setting is backpropagation through the solution of an optimization problem, which often lacks a closed form. This paper provides theoretical insights into the backward pass of unrolled optimization, showing that it is equivalent to the solution of a linear system by a particular iterative method. A system called Folded Optimization is proposed to construct more efficient backpropagation rules from unrolled solver implementations.
  arXiv  Detail & Related papers  (2023-12-28T23:15:18Z)
• Backpropagation of Unrolled Solvers with Folded Optimization [55.04219793298687]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation.
  arXiv  Detail & Related papers  (2023-01-28T01:50:42Z)
• An Empirical Evaluation of Zeroth-Order Optimization Methods on AI-driven Molecule Optimization [78.36413169647408]
  We study the effectiveness of various ZO optimization methods for optimizing molecular objectives. We show the advantages of ZO sign-based gradient descent (ZO-signGD) We demonstrate the potential effectiveness of ZO optimization methods on widely used benchmark tasks from the Guacamol suite.
  arXiv  Detail & Related papers  (2022-10-27T01:58:10Z)
• Neural Combinatorial Optimization: a New Player in the Field [69.23334811890919]
  This paper presents a critical analysis on the incorporation of algorithms based on neural networks into the classical optimization framework. A comprehensive study is carried out to analyse the fundamental aspects of such algorithms, including performance, transferability, computational cost and to larger-sized instances.
  arXiv  Detail & Related papers  (2022-05-03T07:54:56Z)
• First-Order Methods for Convex Optimization [2.578242050187029]
  First-order methods have the potential to provide low accuracy solutions at low computational complexity. We give complete proofs for various key results, and highlight the unifying aspects of several optimization algorithms.
  arXiv  Detail & Related papers  (2021-01-04T13:03:38Z)
• A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning [95.85269649177336]
  ZO optimization iteratively performs three major steps: gradient estimation, descent direction, and solution update. We demonstrate promising applications of ZO optimization, such as evaluating and generating explanations from black-box deep learning models, and efficient online sensor management.
  arXiv  Detail & Related papers  (2020-06-11T06:50:35Z)
• Do optimization methods in deep learning applications matter? [0.0]
  The paper presents arguments on which optimization functions to use and further, which functions would benefit from parallelization efforts. Our experiments compare off-the-shelf optimization functions(CG, SGD, LM and L-BFGS) in standard CIFAR, MNIST, CartPole and FlappyBird experiments.
  arXiv  Detail & Related papers  (2020-02-28T10:36:40Z)
• Self-Directed Online Machine Learning for Topology Optimization [58.920693413667216]
  Self-directed Online Learning Optimization integrates Deep Neural Network (DNN) with Finite Element Method (FEM) calculations. Our algorithm was tested by four types of problems including compliance minimization, fluid-structure optimization, heat transfer enhancement and truss optimization. It reduced the computational time by 2 5 orders of magnitude compared with directly using methods, and outperformed all state-of-the-art algorithms tested in our experiments.
  arXiv  Detail & Related papers  (2020-02-04T20:00:28Z)
• Introduction to Online Convex Optimization [31.771131314017385]
  This manuscript portrays optimization as a process. In many practical applications the environment is so complex that it is infeasible to lay out a comprehensive theoretical model.
  arXiv  Detail & Related papers  (2019-09-07T19:06:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.