Directional Consistency as a Complementary Optimization Signal: The GONO Framework

• URL: http://arxiv.org/abs/2605.06575v1
• Date: Thu, 07 May 2026 17:05:05 GMT
• Title: Directional Consistency as a Complementary Optimization Signal: The GONO Framework
• Abstract summary: We formalize an underexplored phenomenon in deep learning optimization: directional alignment and loss convergence can be decoupled.<n>We introduce GONO, which adapts Adam's momentum beta based on cc_t: amplifying momentum under directional consistency and suppressing it during oscillation.<n>We prove GONO matches Adam's convergence rate and reduces exactly to Adam when the signal is uninformative.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We identify and formalize an underexplored phenomenon in deep learning optimization: directional alignment and loss convergence can be decoupled. An optimizer can exhibit near-perfect directional consistency (cc_t -> 1, measured via consecutive gradient cosine similarity) while the loss remains high or decreases slowly. This observation reveals that existing optimizers such as Adam, SGD, and RMSprop lack explicit mechanisms to exploit temporal consistency in gradient directions, relying instead on magnitude-based signals that fail to distinguish plateaus, saddle points, and genuine convergence. Motivated by this, we introduce GONO (Gradient-Oriented Norm-Adaptive Optimizer), which adapts Adam's momentum coefficient beta_1 based on cc_t: amplifying momentum under directional consistency and suppressing it during oscillation. We prove GONO matches Adam's O(1/sqrt(T)) convergence rate and reduces exactly to Adam when the signal is uninformative. Empirically, cc_t achieves oscillation detection with F1=1.00 (vs. 0.45 for gradient norm), and GONO remains competitive with AdamW on MNIST (98.15%), CIFAR-10 (43.14%), and ResNet-18 (75.44%), establishing directional alignment as a theoretically grounded, practically actionable optimization signal. Code: https://github.com/victordaniel/gono-optimizer

Related papers

• Adaptively Incorporating Directional Hints into Zeroth-Order Optimization [49.940317286021696]
  We develop an algorithm that achieves zeroth-order optimization of non- functions with the aid directional hints.<n>We validate the method, demonstrating sustained progress on non- control tasks.
  arXiv  Detail & Related papers  (2026-09-08T05:42:02Z)
• Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization [8.412194372218895]
  We introduce Singularity-aware Adam (S-Adam), a novel that stabilizes training by dynamically modulating step sizes based on local geometric instability.<n>S-Adam incorporates an adaptive damping mechanism that decelerates updates in high-instability regions while preserving fast convergence in smooth basins.<n> Empirical evaluations on optimal Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD.
  arXiv  Detail & Related papers  (2026-05-28T08:00:40Z)
• FlowAdam: Implicit Regularization via Geometry-Aware Soft Momentum Injection [0.0]
  FlowAdam augments Adam with continuous gradient-flow integration via an ordinary differential equation (ODE)<n>Soft Momentum Injection blends ODE velocity with Adam's momentum during mode transitions.<n>Across coupled optimization benchmarks, the ODE integration provides implicit regularization, reducing held-out error by 10-22%.
  arXiv  Detail & Related papers  (2026-04-08T03:57:52Z)
• Causal Direction from Convergence Time: Faster Training in the True Causal Direction [0.0]
  We introduce Causal Computational Asymmetry (CCA), a principle for causal direction identification based on optimization dynamics.<n>CCA operates in optimization-time space, distinguishing it from methods such as RESIT, IGCI, and SkewScore.<n>We further embed CCA into a broader framework termed Causal Compression Learning (CCL), which integrates graph structure learning, causal information compression, and policy optimization.
  arXiv  Detail & Related papers  (2026-02-24T21:34:57Z)
• Silent Inconsistency in Data-Parallel Full Fine-Tuning: Diagnosing Worker-Level Optimization Misalignment [27.352639822596146]
  Cross-worker divergence in losses and gradients can remain invisible under conventional monitoring signals.<n>We propose a model-agnostic diagnostic framework that quantifies worker-level consistency using training signals readily available in standard pipelines.
  arXiv  Detail & Related papers  (2026-02-16T04:42:30Z)
• Unifying Sign and Magnitude for Optimizing Deep Vision Networks via ThermoLion [0.0]
  Current paradigms impose a static compromise on information channel drift parameters.<n>We introduce a "low-dimensional" exploration model and a "low-dimensional" dynamic alignment framework.
  arXiv  Detail & Related papers  (2025-12-01T17:04:17Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Convergence of Adam Under Relaxed Assumptions [72.24779199744954]
  We show that Adam converges to $epsilon$-stationary points with $O(epsilon-4)$ gradient complexity under far more realistic conditions. We also propose a variance-reduced version of Adam with an accelerated gradient complexity of $O(epsilon-3)$.
  arXiv  Detail & Related papers  (2023-04-27T06:27:37Z)
• Optimizing Mode Connectivity via Neuron Alignment [84.26606622400423]
  Empirically, the local minima of loss functions can be connected by a learned curve in model space along which the loss remains nearly constant. We propose a more general framework to investigate effect of symmetry on landscape connectivity by accounting for the weight permutations of networks being connected.
  arXiv  Detail & Related papers  (2020-09-05T02:25:23Z)
• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning [91.13797346047984]
  We introduce ADAHESSIAN, a second order optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates. We show that ADAHESSIAN achieves new state-of-the-art results by a large margin as compared to other adaptive optimization methods.
  arXiv  Detail & Related papers  (2020-06-01T05:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.