Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy

• URL: http://arxiv.org/abs/2603.13552v1
• Date: Fri, 13 Mar 2026 19:42:12 GMT
• Title: Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy
• Abstract summary: Cross-entropy training analyses rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective.<n>We derive closed-form expressions under logit linearization along the proposed update direction.<n>Temperature scaling confirms the mechanism: normalizing by $_a$ shrinks the onset-threshold spread from standard deviation $0.992$ to $0.164$.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optimization analyses for cross-entropy training rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective. These surrogates are reliable only inside the Taylor convergence radius of the true loss along the update direction. That radius is set not by real-line curvature alone but by the nearest complex singularity. For cross-entropy, the softmax partition function $F=\sum_j \exp(z_j)$ has complex zeros -- ``ghosts of softmax'' -- that induce logarithmic singularities in the loss and cap this radius. To make this geometry usable, we derive closed-form expressions under logit linearization along the proposed update direction. In the binary case, the exact radius is $ρ^*=\sqrt{δ^2+ π^2}/Δ_a$. In the multiclass case, we obtain the lower bound $ρ_a=π/Δ_a$, where $Δ_a=\max_k a_k-\min_k a_k$ is the spread of directional logit derivatives $a_k=\nabla z_k\cdot v$. This bound costs one Jacobian-vector product and reveals what makes a step fragile: samples that are both near a decision flip and highly sensitive to the proposed direction tighten the radius. The normalized step size $r=τ/ρ_a$ separates safe from dangerous updates. Across six tested architectures and multiple step directions, no model fails for $r<1$, yet collapse appears once $r\ge 1$. Temperature scaling confirms the mechanism: normalizing by $ρ_a$ shrinks the onset-threshold spread from standard deviation $0.992$ to $0.164$. A controller that enforces $τ\leρ_a$ survives learning-rate spikes up to $10{,} 000\times$ in our tests, where gradient clipping still collapses. Together, these results identify a geometric constraint on cross-entropy optimization that operates through Taylor convergence rather than Hessian curvature.

Related papers

• One Inverse Step is a Convex Program: Bayes-Limit Calibration of Diffusion Inversion [0.0]
  One implicit DDIM inversion step is the cheapest probe of whether a pretrained diffusion model encodes local manifold geometry.<n>The Fermi window conflicts with the model's own training support by $3.6$-$5.6times$, and the trained Hessian-Lipschitz constant is $2$-$12%$ of the curvature the law reads.<n>Finally the unconditional ceiling $_t_max(mathrmsym,J)le1$, from $mathrmCov(x_0mid x_t)succeq0$
  arXiv  Detail & Related papers  (2026-08-24T11:01:54Z)
• Scaling Limits of Constant-Stepsize SGD at Flat Minima [5.608222858044445]
  We study Markovian noise generated by a contractive driving chain.<n>For gradient descent (SGD) with a constant stepsize $$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons.<n>We show that this behavior changes fundamentally for convex objectives $H$ with flat minima and (sub)quadratic tails.
  arXiv  Detail & Related papers  (2026-07-17T17:22:46Z)
• Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design [28.74829605004241]
  We characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss)<n>We show that GD linearly converges to a small neighborhood of $*$ achieving an $ell$ error of order $O(sqrt|*|5d/n)$.<n>We also establish a faster local linear convergence to the same statistical error under a large $(|*|_2)$ stepsize.
  arXiv  Detail & Related papers  (2026-06-19T18:43:41Z)
• When Does $\ell_2$-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the $\ell_1$ Implicit Bias [15.113649527486276]
  We show that benign overfitting fails at the linear rate.<n>We show that while this localization mechanism should persist in the presence of signals, the exact signal-noise decomposition remains an open problem.
  arXiv  Detail & Related papers  (2026-05-07T14:14:09Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• Closed-form $\ell_r$ norm scaling with data for overparameterized linear regression and diagonal linear networks under $\ell_p$ bias [0.0]
  We give a unified, high-probability characterization for the scaling of the family of parameter norms.<n>We then study linear networks trained by descent.
  arXiv  Detail & Related papers  (2025-09-25T13:59:22Z)
• Proving the Limited Scalability of Centralized Distributed Optimization via a New Lower Bound Construction [57.93371273485736]
  We consider a centralized distributed learning setup where all workers jointly find an unbiased bound LDeltaepsilon2,$ better poly-logarithmically in $n$, even in the homogeneous (i.i.d.) case, where all workers access the same distribution.
  arXiv  Detail & Related papers  (2025-06-30T13:27:39Z)
• A Proximal Modified Quasi-Newton Method for Nonsmooth Regularized Optimization [0.7373617024876725]
  Lipschitz-of-$nabla f$. $mathcalS_k|p$. $mathcalS_k|p$. $nabla f$. $mathcalS_k|p$.
  arXiv  Detail & Related papers  (2024-09-28T18:16:32Z)
• Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization [18.47705532817026]
  Adaptive gradient methods are among the most successful neural network training algorithms.<n>These methods are known to achieve better dimensional dependence than over convex SGD-normarity.<n>In this paper we introduce new assumptions on the smoothness of the structure and the gradient noise variance.
  arXiv  Detail & Related papers  (2024-06-07T02:55:57Z)
• A Unified Framework for Uniform Signal Recovery in Nonlinear Generative Compressed Sensing [68.80803866919123]
  Under nonlinear measurements, most prior results are non-uniform, i.e., they hold with high probability for a fixed $mathbfx*$ rather than for all $mathbfx*$ simultaneously. Our framework accommodates GCS with 1-bit/uniformly quantized observations and single index models as canonical examples. We also develop a concentration inequality that produces tighter bounds for product processes whose index sets have low metric entropy.
  arXiv  Detail & Related papers  (2023-09-25T17:54:19Z)
• Measurement-induced phase transition for free fermions above one dimension [46.176861415532095]
  Theory of the measurement-induced entanglement phase transition for free-fermion models in $d>1$ dimensions is developed. Critical point separates a gapless phase with $elld-1 ln ell$ scaling of the second cumulant of the particle number and of the entanglement entropy.
  arXiv  Detail & Related papers  (2023-09-21T18:11:04Z)
• ReSQueing Parallel and Private Stochastic Convex Optimization [59.53297063174519]
  We introduce a new tool for BFG convex optimization (SCO): a Reweighted Query (ReSQue) estimator for the gradient of a function convolved with a (Gaussian) probability density. We develop algorithms achieving state-of-the-art complexities for SCO in parallel and private settings.
  arXiv  Detail & Related papers  (2023-01-01T18:51:29Z)
• Optimal Robust Linear Regression in Nearly Linear Time [97.11565882347772]
  We study the problem of high-dimensional robust linear regression where a learner is given access to $n$ samples from the generative model $Y = langle X,w* rangle + epsilon$ We propose estimators for this problem under two settings: (i) $X$ is L4-L2 hypercontractive, $mathbbE [XXtop]$ has bounded condition number and $epsilon$ has bounded variance and (ii) $X$ is sub-Gaussian with identity second moment and $epsilon$ is
  arXiv  Detail & Related papers  (2020-07-16T06:44:44Z)
• How isotropic kernels perform on simple invariants [0.5729426778193397]
  We investigate how the training curve of isotropic kernel methods depends on the symmetry of the task to be learned. We show that for large bandwidth, $beta = fracd-1+xi3d-3+xi$, where $xiin (0,2)$ is the exponent characterizing the stripe of the kernel at the origin.
  arXiv  Detail & Related papers  (2020-06-17T09:59:18Z)
• Agnostic Learning of a Single Neuron with Gradient Descent [92.7662890047311]
  We consider the problem of learning the best-fitting single neuron as measured by the expected square loss. For the ReLU activation, our population risk guarantee is $O(mathsfOPT1/2)+epsilon$. For the ReLU activation, our population risk guarantee is $O(mathsfOPT1/2)+epsilon$.
  arXiv  Detail & Related papers  (2020-05-29T07:20:35Z)
• Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness [151.67113334248464]
  We show that extending the smoothing technique to defend against other attack models can be challenging. We present experimental results on CIFAR to validate our theory.
  arXiv  Detail & Related papers  (2020-02-08T22:02:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.