Evolving Deep Learning Optimizers

• URL: http://arxiv.org/abs/2512.11853v1
• Date: Fri, 05 Dec 2025 01:30:07 GMT
• Title: Evolving Deep Learning Optimizers
• Abstract summary: We present a genetic algorithm framework for discovering deep learning optimization algorithms.<n>Our approach encodes genomes that specify combinations of primitive update terms.<n>We find an evolved that outperforms Adam by 2.6% in aggregate fitness.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a genetic algorithm framework for automatically discovering deep learning optimization algorithms. Our approach encodes optimizers as genomes that specify combinations of primitive update terms (gradient, momentum, RMS normalization, Adam-style adaptive terms, and sign-based updates) along with hyperparameters and scheduling options. Through evolutionary search over 50 generations with a population of 50 individuals, evaluated across multiple vision tasks, we discover an evolved optimizer that outperforms Adam by 2.6% in aggregate fitness and achieves a 7.7% relative improvement on CIFAR-10. The evolved optimizer combines sign-based gradient terms with adaptive moment estimation, uses lower momentum coefficients than Adam ($β_1$=0.86, $β_2$=0.94), and notably disables bias correction while enabling learning rate warmup and cosine decay. Our results demonstrate that evolutionary search can discover competitive optimization algorithms and reveal design principles that differ from hand-crafted optimizers. Code is available at https://github.com/mmarfinetz/evo-optimizer.

Related papers

• GEGO: A Hybrid Golden Eagle and Genetic Optimization Algorithm for Efficient Hyperparameter Tuning in Resource-Constrained Environments [4.417564179511245]
  HyperEGO tuning is a critical yet computationally expensive step in hyper search networks.<n>Golden Eagle Optimization (GEGO) integrates a hybrid movement strategy of Golden Eagle operators of selection and mutation.<n>GEGO is evaluated on standard unimodal, multimodal, and composite benchmark functions.
  arXiv  Detail & Related papers  (2026-01-21T05:35:38Z)
• Quantum Time-Series Learning with Evolutionary Algorithms [0.0]
  Variational quantum circuits have arisen as an important method in quantum computing.<n>We explore the use of evolutionary algorithms for such optimization, specifically for time-series forecasting.
  arXiv  Detail & Related papers  (2024-12-23T13:53:35Z)
• MARS: Unleashing the Power of Variance Reduction for Training Large Models [56.67982828148859]
  We propose a unified training framework for deep neural networks.<n>We introduce three instances of MARS that leverage preconditioned gradient optimization.<n>Results indicate that the implementation of MARS consistently outperforms Adam.
  arXiv  Detail & Related papers  (2024-11-15T18:57:39Z)
• Adaptive Friction in Deep Learning: Enhancing Optimizers with Sigmoid and Tanh Function [0.0]
  We introduce sigSignGrad and tanhSignGrad, two novel gradients that integrate adaptive friction coefficients. Our theoretical analysis demonstrates the wide-ranging adjustment capability of the friction coefficient S. Experiments on CIFAR-10, Mini-Image-Net using ResNet50 and ViT architectures confirm the superior performance our proposeds.
  arXiv  Detail & Related papers  (2024-08-07T03:20:46Z)
• Frog-Snake prey-predation Relationship Optimization (FSRO) : A novel nature-inspired metaheuristic algorithm for feature selection [0.0]
  This study proposes the Frog-Snake prey-predation Relationship Optimization (FSRO) algorithm. It is inspired by the prey-predation relationship between frogs and snakes for application to discrete optimization problems. The proposed algorithm conducts computational experiments on feature selection using 26 types of machine learning datasets.
  arXiv  Detail & Related papers  (2024-02-13T06:39:15Z)
• MADA: Meta-Adaptive Optimizers through hyper-gradient Descent [73.1383658672682]
  We introduce Meta-Adaptives (MADA), a unified framework that can generalize several known convergences and dynamically learn the most suitable one during training. We empirically compare MADA to other populars on vision and language tasks, and find that MADA consistently outperforms Adam and other populars. We also propose AVGrad, a modification of AMS that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization.
  arXiv  Detail & Related papers  (2024-01-17T00:16:46Z)
• Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers [109.52244418498974]
  We propose a novel textscAdmeta (textbfADouble exponential textbfMov averagtextbfE textbfAdaptive and non-adaptive momentum) framework. We provide two implementations, textscAdmetaR and textscAdmetaS, the former based on RAdam and the latter based on SGDM.
  arXiv  Detail & Related papers  (2023-07-02T18:16:06Z)
• Massively Parallel Genetic Optimization through Asynchronous Propagation of Populations [50.591267188664666]
  Propulate is an evolutionary optimization algorithm and software package for global optimization. We provide an MPI-based implementation of our algorithm, which features variants of selection, mutation, crossover, and migration. We find that Propulate is up to three orders of magnitude faster without sacrificing solution accuracy.
  arXiv  Detail & Related papers  (2023-01-20T18:17:34Z)
• An efficient hybrid classification approach for COVID-19 based on Harris Hawks Optimization and Salp Swarm Optimization [0.0]
  This study presents a hybrid binary version of the Harris Hawks Optimization algorithm (HHO) and Salp Swarm Optimization (SSA) for Covid-19 classification. The proposed algorithm (HHOSSA) achieved 96% accuracy with the SVM, 98% and 98% accuracy with two classifiers.
  arXiv  Detail & Related papers  (2022-12-25T19:52:18Z)
• VeLO: Training Versatile Learned Optimizers by Scaling Up [67.90237498659397]
  We leverage the same scaling approach behind the success of deep learning to learn versatiles. We train an ingest for deep learning which is itself a small neural network that ingests and outputs parameter updates. We open source our learned, meta-training code, the associated train test data, and an extensive benchmark suite with baselines at velo-code.io.
  arXiv  Detail & Related papers  (2022-11-17T18:39:07Z)
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks [21.63353575405414]
  We study the implicit bias of adaptive optimization algorithms on homogeneous neural networks. It is the first work to study the convergent direction of adaptive optimizations on non-linear deep neural networks.
  arXiv  Detail & Related papers  (2020-12-11T11:15:32Z)
• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning [91.13797346047984]
  We introduce ADAHESSIAN, a second order optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates. We show that ADAHESSIAN achieves new state-of-the-art results by a large margin as compared to other adaptive optimization methods.
  arXiv  Detail & Related papers  (2020-06-01T05:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.