Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective

• URL: http://arxiv.org/abs/2605.03373v1
• Date: Tue, 05 May 2026 05:28:29 GMT
• Title: Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective
• Abstract summary: A growing body of recent work demonstrates the successful application of ZO methods to fine-tuning Large Language Models.<n>We derive the one-step learning dynamics of ZO SGD, where the empirical Neural Tangent Kernel (eNTK) emerges as the key term governing the learning behavior.<n>We believe that this kernel-based framework offers a novel perspective for understanding ZO methods within the context of learning dynamics.
• Score: 8.18887614959892
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical optimization theory establishes that zeroth-order (ZO) algorithms suffer from a dimension-dependent slowdown, with convergence rates typically scaling with the model dimension compared to first-order methods. However, in contrast to these theoretical expectations, a growing body of recent work demonstrates the successful application of ZO methods to fine-tuning Large Language Models (LLMs) with billions of parameters. To explain this paradox, we derive the one-step learning dynamics of ZO SGD, where the empirical Neural Tangent Kernel (eNTK) naturally emerges as the key term governing the learning behavior. Inspection of the eNTK produced by ZO SGD reveals that each element corresponds to the inner product of neural tangent vectors projected onto a random low-dimensional subspace. Thus, by invoking the Johnson-Lindenstrauss Lemma, our analysis shows that the fidelity of the ZO eNTK is governed primarily by the number of perturbations. Crucially, the approximation error depends on the model output size rather than the massive parameter dimension. This dimension-free property provides a theoretical justification for the scalability of ZO methods to LLMs finetuning tasks. We believe that this kernel-based framework offers a novel perspective for understanding ZO methods within the context of learning dynamics.

Related papers

• Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods [42.41448048632761]
  This paper provides a comprehensive generalization analysis of deep neural networks (DNNs) trained using gradient-based methods.<n>We establish, for the first time, a crucial connection between the learning dynamics of a DNN with smooth activation functions trained via gradient-based methods and those of kernel methods.<n>We derive the first known minimax-optimal rates for the excess population risk of gradient descent (GD) and gradient descent (SGD)
  arXiv  Detail & Related papers  (2026-06-04T23:31:52Z)
• Model Evolution Under Zeroth-Order Optimization: A Neural Tangent Kernel Perspective [33.99966177443865]
  We introduce the Neural Zeroth-order Kernel (NZK) to describe model evolution in function space under ZO updates.<n>We prove that the expected NZK remains constant throughout training and depends explicitly on the first and second moments of the random perturbation directions.
  arXiv  Detail & Related papers  (2026-03-22T11:14:53Z)
• Deriving Hyperparameter Scaling Laws via Modern Optimization Theory [55.63126290312615]
  We study modern first-order boundss for methods based on the Linear Minimization Oracle (LMO)<n>Treating bounds in recent literature as a proxy and minimizing them across different tuning regimes yields closed-form power-law schedules for learning rate, momentum, and batch size.<n>Our results draw particular attention to the interaction between momentum and batch-size scaling, suggesting that optimal performance may be achieved with several scaling strategies.
  arXiv  Detail & Related papers  (2026-03-16T22:21:27Z)
• A Geometrically-Grounded Drive for MDL-Based Optimization in Deep Learning [3.2452107817263003]
  This paper introduces a novel optimization framework that fundamentally integrates the Minimum Description Length (MDL) principle into the training dynamics of deep neural networks.<n>We provide a practical, computationally efficient algorithm with $O(N log N)$ per-iteration complexity (Theoremrefthm:complexity), alongside guarantees for numerical stability (Theoremrefthm:stability) and exponential convergence under convexity assumptions (Theoremrefthm:convergence_rate)<n>This work provides a principled path toward more autonomous, generalizable, and interpretable AI
  arXiv  Detail & Related papers  (2026-03-12T08:31:00Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Gradient Descent as a Perceptron Algorithm: Understanding Dynamics and Implicit Acceleration [67.12978375116599]
  We show that the steps of gradient descent (GD) reduce to those of generalized perceptron algorithms.<n>This helps explain the optimization dynamics and the implicit acceleration phenomenon observed in neural networks.
  arXiv  Detail & Related papers  (2025-12-12T14:16:35Z)
• A joint optimization approach to identifying sparse dynamics using least squares kernel collocation [70.13783231186183]
  We develop an all-at-once modeling framework for learning systems of ordinary differential equations (ODE) from scarce, partial, and noisy observations of the states.<n>The proposed methodology amounts to a combination of sparse recovery strategies for the ODE over a function library combined with techniques from reproducing kernel Hilbert space (RKHS) theory for estimating the state and discretizing the ODE.
  arXiv  Detail & Related papers  (2025-11-23T18:04:15Z)
• Beyond the Ideal: Analyzing the Inexact Muon Update [54.70108543057578]
  We show first analysis of the inexactized update at Muon's core.<n>We reveal a fundamental coupling between this inexactness and the optimal step size and momentum.
  arXiv  Detail & Related papers  (2025-10-22T18:01:07Z)
• Modeling Latent Neural Dynamics with Gaussian Process Switching Linear Dynamical Systems [2.170477444239546]
  We develop an approach that balances these two objectives: the Gaussian Process Switching Linear Dynamical System (gpSLDS)<n>Our method builds on previous work modeling the latent state evolution via a differential equation whose nonlinear dynamics are described by a Gaussian process (GP-SDEs)<n>Our approach resolves key limitations of the rSLDS such as artifactual oscillations in dynamics near discrete state boundaries, while also providing posterior uncertainty estimates of the dynamics.
  arXiv  Detail & Related papers  (2024-07-19T15:32:15Z)
• A Deep Dive into the Connections Between the Renormalization Group and Deep Learning in the Ising Model [0.0]
  Renormalization group (RG) is an essential technique in statistical physics and quantum field theory. We develop extensive renormalization techniques for the 1D and 2D Ising model to provide a baseline for comparison. For the 2D Ising model, we successfully generated Ising model samples using the Wolff algorithm, and performed the group flow using a quasi-deterministic method.
  arXiv  Detail & Related papers  (2023-08-21T22:50:54Z)
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression [53.15502562048627]
  Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator. This work delves into a statistical analysis of augmentation-based pretraining.
  arXiv  Detail & Related papers  (2023-06-01T15:18:55Z)
• Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis [7.373617024876726]
  Several non-linear operators in analysis depend on a temporal structure which is not leveraged by contemporary neural operators.<n>This paper introduces a deep learning model-design framework that takes suitable infinite-dimensional linear metric spaces.<n>We show that our framework can uniformly approximate on compact sets and across arbitrarily finite-time horizons H" or smooth trace class operators.
  arXiv  Detail & Related papers  (2022-10-24T14:43:03Z)
• An Ode to an ODE [78.97367880223254]
  We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d) This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
  arXiv  Detail & Related papers  (2020-06-19T22:05:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.