Dynamically Weighted Momentum with Adaptive Step Sizes for Efficient Deep Network Training

• URL: http://arxiv.org/abs/2510.25042v1
• Date: Wed, 29 Oct 2025 00:03:03 GMT
• Title: Dynamically Weighted Momentum with Adaptive Step Sizes for Efficient Deep Network Training
• Authors: Zhifeng Wang, Longlong Li, Chunyan Zeng,
• Abstract summary: This paper introduces a novel deep learning algorithm named DWM DWMGrad.<n>It incorporates a dynamic mechanism reliant on historical data to dynamically update momentum learning rates.
• Score: 6.320135812353531
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Within the current sphere of deep learning research, despite the extensive application of optimization algorithms such as Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), there remains a pronounced inadequacy in their capability to address fluctuations in learning efficiency, meet the demands of complex models, and tackle non-convex optimization issues. These challenges primarily arise from the algorithms' limitations in handling complex data structures and models, for instance, difficulties in selecting an appropriate learning rate, avoiding local optima, and navigating through high-dimensional spaces. To address these issues, this paper introduces a novel optimization algorithm named DWMGrad. This algorithm, building on the foundations of traditional methods, incorporates a dynamic guidance mechanism reliant on historical data to dynamically update momentum and learning rates. This allows the optimizer to flexibly adjust its reliance on historical information, adapting to various training scenarios. This strategy not only enables the optimizer to better adapt to changing environments and task complexities but also, as validated through extensive experimentation, demonstrates DWMGrad's ability to achieve faster convergence rates and higher accuracies under a multitude of scenarios.

Related papers

• Context-Aware Rule Mining Using a Dynamic Transformer-Based Framework [8.52080590054588]
  This study proposes a dynamic rule data mining algorithm based on an improved Transformer architecture.<n>We show that the improved Transformer model has achieved significant improvements in rule mining accuracy, coverage, and stability.<n>Future research will focus on optimizing computational efficiency and combining more deep learning technologies to expand the application scope of the algorithm.
  arXiv  Detail & Related papers  (2025-03-14T06:37:04Z)
• Integrating Optimization Theory with Deep Learning for Wireless Network Design [38.257335693563554]
  Traditional wireless network design relies on optimization algorithms derived from domain-specific mathematical models.<n>Deep learning has emerged as a promising alternative to overcome complexity and adaptability concerns.<n>This paper introduces a novel approach that integrates optimization theory with deep learning methodologies to address these issues.
  arXiv  Detail & Related papers  (2024-12-11T20:27:48Z)
• Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws [59.03420759554073]
  We introduce Adaptive Data Optimization (ADO), an algorithm that optimize data distributions in an online fashion, concurrent with model training. ADO does not require external knowledge, proxy models, or modifications to the model update. ADO uses per-domain scaling laws to estimate the learning potential of each domain during training and adjusts the data mixture accordingly.
  arXiv  Detail & Related papers  (2024-10-15T17:47:44Z)
• Memory-Efficient Optimization with Factorized Hamiltonian Descent [11.01832755213396]
  We introduce a novel adaptive, H-Fac, which incorporates a memory-efficient factorization approach to address this challenge. By employing a rank-1 parameterization for both momentum and scaling parameter estimators, H-Fac reduces memory costs to a sublinear level. We develop our algorithms based on principles derived from Hamiltonian dynamics, providing robust theoretical underpinnings in optimization dynamics and convergence guarantees.
  arXiv  Detail & Related papers  (2024-06-14T12:05:17Z)
• Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation [20.851925464903804]
  This paper introduces a novel learning paradigm, Dynamic Sparse Learning, tailored for recommendation models. DSL innovatively trains a lightweight sparse model from scratch, periodically evaluating and dynamically adjusting each weight's significance. Our experimental results underline DSL's effectiveness, significantly reducing training and inference costs while delivering comparable recommendation performance.
  arXiv  Detail & Related papers  (2024-02-05T10:16:20Z)
• Multiplicative update rules for accelerating deep learning training and increasing robustness [69.90473612073767]
  We propose an optimization framework that fits to a wide range of machine learning algorithms and enables one to apply alternative update rules. We claim that the proposed framework accelerates training, while leading to more robust models in contrast to traditionally used additive update rule.
  arXiv  Detail & Related papers  (2023-07-14T06:44:43Z)
• A Data-Driven Evolutionary Transfer Optimization for Expensive Problems in Dynamic Environments [9.098403098464704]
  Data-driven, a.k.a. surrogate-assisted, evolutionary optimization has been recognized as an effective approach for tackling expensive black-box optimization problems. This paper proposes a simple but effective transfer learning framework to empower data-driven evolutionary optimization to solve dynamic optimization problems. Experiments on synthetic benchmark test problems and a real-world case study demonstrate the effectiveness of our proposed algorithm.
  arXiv  Detail & Related papers  (2022-11-05T11:19:50Z)
• Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspective [52.497514255040514]
  This work develops a new approach that enables data-driven methods to continuously learn and optimize resource allocation strategies in a dynamic environment. We propose to build the notion of continual learning into wireless system design, so that the learning model can incrementally adapt to the new episodes. Our design is based on a novel bilevel optimization formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2021-05-03T07:23:39Z)
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment [55.91291559442884]
  This work develops a methodology that enables data-driven methods to continuously learn and optimize in a dynamic environment. We propose to build the notion of continual learning into the modeling process of learning wireless systems. Our design is based on a novel min-max formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2020-11-16T08:24:34Z)
• Optimization-driven Machine Learning for Intelligent Reflecting Surfaces Assisted Wireless Networks [82.33619654835348]
  Intelligent surface (IRS) has been employed to reshape the wireless channels by controlling individual scattering elements' phase shifts. Due to the large size of scattering elements, the passive beamforming is typically challenged by the high computational complexity. In this article, we focus on machine learning (ML) approaches for performance in IRS-assisted wireless networks.
  arXiv  Detail & Related papers  (2020-08-29T08:39:43Z)
• Automatically Learning Compact Quality-aware Surrogates for Optimization Problems [55.94450542785096]
  Solving optimization problems with unknown parameters requires learning a predictive model to predict the values of the unknown parameters and then solving the problem using these values. Recent work has shown that including the optimization problem as a layer in a complex training model pipeline results in predictions of iteration of unobserved decision making. We show that we can improve solution quality by learning a low-dimensional surrogate model of a large optimization problem.
  arXiv  Detail & Related papers  (2020-06-18T19:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.