Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts

• URL: http://arxiv.org/abs/2603.10095v1
• Date: Tue, 10 Mar 2026 16:31:02 GMT
• Title: Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts
• Abstract summary: Time-series forecasting often faces challenges from distributional drift.<n>This dynamic behavior can undermine the effectiveness of adaptive landscapes, such as Adam.<n>We propose TS_Adam, a lightweight variant that removes the second-order correction from the learning rate.
• Score: 1.8022002874089529
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Time-series forecasting often faces challenges from non-stationarity, particularly distributional drift, where the data distribution evolves over time. This dynamic behavior can undermine the effectiveness of adaptive optimizers, such as Adam, which are typically designed for stationary objectives. In this paper, we revisit Adam in the context of non-stationary forecasting and identify that its second-order bias correction limits responsiveness to shifting loss landscapes. To address this, we propose TS_Adam, a lightweight variant that removes the second-order correction from the learning rate computation. This simple modification improves adaptability to distributional drift while preserving the optimizer core structure and requiring no additional hyperparameters. TS_Adam integrates easily into existing models and consistently improves performance across long- and short-term forecasting tasks. On the ETT datasets with the MICN model, it achieves an average reduction of 12.8% in MSE and 5.7% in MAE compared to Adam. These results underscore the practicality and versatility of TS_Adam as an effective optimization strategy for real-world forecasting scenarios involving non-stationary data. Code is available at: https://github.com/DD-459-1/TS_Adam.

Related papers

• Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization [8.412194372218895]
  We introduce Singularity-aware Adam (S-Adam), a novel that stabilizes training by dynamically modulating step sizes based on local geometric instability.<n>S-Adam incorporates an adaptive damping mechanism that decelerates updates in high-instability regions while preserving fast convergence in smooth basins.<n> Empirical evaluations on optimal Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD.
  arXiv  Detail & Related papers  (2026-05-28T08:00:40Z)
• Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates [13.40796672049436]
  This paper proposes Ada2MS, an optimization algorithm that achieves a smooth transition between AdamW-like behavior and momentum-SGD-like behavior.<n>On the visual tasks evaluated in this study, Ada2MS obtains competitive results under a unified-comparison protocol.
  arXiv  Detail & Related papers  (2026-05-19T22:11:04Z)
• AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate [13.40796672049436]
  The AdamNX algorithm is proposed to converge high-dimensional optimization to local and even global minima.<n>Its core innovation lies in the proposition of a novel type of second-order moment estimation exponential decay rate.<n>Our results show that our second-order moment estimation exponential decay rate is better than the current second-order moment estimation exponential decay rate.
  arXiv  Detail & Related papers  (2025-11-17T15:07:55Z)
• AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training [22.58304858379219]
  We introduce AdamS, a simple yet effective alternative to Adam for large language model (LLM) pretraining and post-training.<n>By leveraging a novel denominator, i.e., the root of weighted sum of squares of the momentum and the current gradient, AdamS eliminates the need for second-moment estimates.<n>AdamS is efficient, matching the memory and compute footprint of SGD with momentum while delivering superior optimization performance.
  arXiv  Detail & Related papers  (2025-05-22T08:16:48Z)
• Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps [65.64965527170156]
  We adapt the widely used Adam optimiser for use in reinforcement learning.<n>We show that Adam-Rel uses the local timestep within an epoch, essentially resetting Adam's timestep to 0 after target changes.<n>We then show that increases in gradient norm occur in RL in practice, and examine the differences between our theoretical model and the observed data.
  arXiv  Detail & Related papers  (2024-12-22T18:01:08Z)
• CAdam: Confidence-Based Optimization for Online Learning [41.022196390765714]
  We introduce CAdam, a confidence-based optimization strategy that assesses the consistency between the momentum and the gradient for each parameter dimension before deciding on updates.<n>CAdam significantly enhances model performance compared to Adam, leading to substantial increases in the system's gross merchandise volume (GMV)<n>In large-scale A/B testing, CAdam significantly enhances model performance compared to Adam, leading to substantial increases in the system's gross merchandise volume (GMV)
  arXiv  Detail & Related papers  (2024-11-29T12:00:27Z)
• MARS: Unleashing the Power of Variance Reduction for Training Large Models [56.67982828148859]
  We propose a unified training framework for deep neural networks.<n>We introduce three instances of MARS that leverage preconditioned gradient optimization.<n>Results indicate that the implementation of MARS consistently outperforms Adam.
  arXiv  Detail & Related papers  (2024-11-15T18:57:39Z)
• Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models [68.23649978697027]
  Forecast-PEFT is a fine-tuning strategy that freezes the majority of the model's parameters, focusing adjustments on newly introduced prompts and adapters. Our experiments show that Forecast-PEFT outperforms traditional full fine-tuning methods in motion prediction tasks. Forecast-FT further improves prediction performance, evidencing up to a 9.6% enhancement over conventional baseline methods.
  arXiv  Detail & Related papers  (2024-07-28T19:18:59Z)
• Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization [90.08459757321405]
  Federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead.<n>We propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates local model parameters and moment estimates.<n>By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error.
  arXiv  Detail & Related papers  (2024-05-28T07:56:49Z)
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients [112.00379151834242]
  We propose adaptive learning rate principle, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance each coordinate. This results in faster adaptation, which leads more desirable empirical convergence behaviors.
  arXiv  Detail & Related papers  (2020-06-21T21:47:43Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.