Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

• URL: http://arxiv.org/abs/2607.22474v1
• Date: Fri, 24 Jul 2026 16:42:46 GMT
• Title: Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent
• Abstract summary: We show that negative-shifted gradient descent escapes a structural constraint.<n>The filter is smooth at the would-be pole and mixed-sign-capable.<n>A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.
• Score: 11.327284468812948
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In overparameterized linear regression, many weak spectral directions act like a ridge penalty on the signal-bearing spectrum; negative ridge is the natural correction, pushing filters above one. The stable negative-ridge endpoint, however, is structurally limited: its pole must stay below the smallest nonzero empirical eigenvalue, and it anti-shrinks smaller eigenvalues more than larger ones. Early-stopped negative-shifted gradient descent escapes this constraint. Its filter is smooth at the would-be pole and mixed-sign-capable: above-ridgeless directions form a leading prefix, with lower directions shrunk or exposure-controlled while stopping sets the crossover. In a Gaussian spike-plus-flat model we discover a Marchenko-Pastur barrier: the shift that cancels the implicit penalty lies a bulk width above the smallest empirical eigenvalue, and the stopped path improves on every admissible endpoint by a polynomial factor in risk under explicit conditions. Our main theorem permits a general high-effective-rank tail: its trace sets the implicit floor, its squared spectrum controls exposure, and the floor-critical path recovers all head scales at once, beyond positive shrinkage and, once scales separate, every uniform rescaling of ridgeless. Handling the noncontractive shifted dynamics is the central technical challenge; localized Duhamel integrals control them. A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.

Related papers

• Constrained Online Learning with Noisy Constraint Values [55.29259818039367]
  Under common feasibility, our LEDGER algorithm attains $O(sqrt T)$ expected regret and $O(sqrtTlog(eT))$ expected budget violation.<n>Neither a Slater condition, independence between feedback channels, nor an absolute constraint-value bound is needed.
  arXiv  Detail & Related papers  (2026-09-07T01:38:41Z)
• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Interior skin focusing and directional mirror transfer in a graded non-Hermitian Krawtchouk network [0.0]
  We study a finite open Krawtchouk network with oppositely graded directed hoppings.<n>For a broad intermediate range of asymmetries, their local imaginary gauge field changes sign in the bulk.
  arXiv  Detail & Related papers  (2026-08-19T10:56:53Z)
• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data [3.7437735326653883]
  This paper studies the rate-controlled middle case for full-batch linear classification on separable data.<n>For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice.
  arXiv  Detail & Related papers  (2026-07-07T08:55:35Z)
• Orthogonality Edges in Strong-Coupling Quantum Work Statistics [0.5541461456285174]
  Strong inversion to a reservoir can do more than shift, broaden, or dress the work peaks of a driven quantum system.<n>We demonstrate this mechanism for the inclusive work distribution of the biased spin-boson model under a sudden bias inversion.
  arXiv  Detail & Related papers  (2026-07-04T16:53:05Z)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory [23.912005398880293]
  Continual learning requires new-task adaptation without damaging previously acquired capabilities.<n>Recent forward-pass and zeroth-order (ZO) results show that low-query adaptation may retain better than first-order descent.<n>We give a local randomized gradient-shaping analysis: finite differences expose a raw shape that is mean-aligned with FO.<n>For norm-matched ZO, the expected shaped retention curvature obeys an exact identity that preserves the isotropic retention floor while contracting only the anisotropic component.<n>Projecting this identity onto the incoming gradient yields the observable FO--ZO quadratic forgetting gap.
  arXiv  Detail & Related papers  (2026-05-11T14:41:08Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• Topological Discrimination of Steep to Supersteep Gap as Evidence of Tunneling in Adiabatic Quantum Processes [0.0]
  It is shown that the gap that limits the speed of a quantum annealing process can take three salient morphologies. The position of the swallow tails relative to the boundary allows the supersteep versus steep discrimination. The absence of swallow tail-boundary interaction characterizes the mild gap.
  arXiv  Detail & Related papers  (2023-11-17T05:34:08Z)
• Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion [83.90492831583997]
  We show that a batch-normalized network can keep the optimal signal propagation properties, but avoid exploding gradients in depth. We use a Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded depth. We also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.
  arXiv  Detail & Related papers  (2023-10-03T12:35:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.