Branch Geometry and Finite-Radius Sensitivity of Hard-ReLU Training

• URL: http://arxiv.org/abs/2608.30960v3
• Date: Sun, 06 Sep 2026 13:07:51 GMT
• Title: Branch Geometry and Finite-Radius Sensitivity of Hard-ReLU Training
• Abstract summary: Outer-learning algorithms use infinitesimal sensitivities to propose finite changes to training parameters.<n>For hard-ReLU training, the derivative of the finite program and the derivative of its flow limit do not by themselves specify the response at a chosen radius.<n>We characterize the intervening regime in which the radius is proportional to the GD step event rounding then survives at leading order.
• Score: 2.969260589680338
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Outer-learning algorithms use infinitesimal sensitivities to propose finite changes to initialization or training parameters. For hard-ReLU training, the derivative of the finite program and the derivative of its flow limit do not by themselves specify the response at a chosen radius. We characterize the intervening regime in which the perturbation radius is proportional to the GD step. Integer event rounding then survives at leading order: smooth Euler bias shifts each discrete phase, and upstream rounding moves downstream branch boundaries. We derive the crossing indices and a uniform endpoint expansion for finitely many separated transverse events in piecewise-$C^2$ dynamics, away from recursive phase boundaries. In contractive affine regions, an explicit remainder and complete branch verification certify finite candidate comparisons. Scalar phase frequencies and a coupled feedback ablation test the mechanism; frozen nonlinear-network experiments show radius-dependent prediction accuracy, including incomplete branch matches and failed-word tails. Together with local AD and uniform flow consistency, the result identifies sufficient response regimes: differentiating training is a choice of perturbation resolution as well as a choice of derivative.

Related papers

• Linearized PINN with pretrained nonlinear layers [48.01541150431857]
  lPINN is a reduced-order neural basis method for forward and inverse differential equations.<n>We evaluate lPINN on forward and inverse problems for the advection-diffusion equation, Burgers' equation, and the nonlinear pendulum equation.
  arXiv  Detail & Related papers  (2026-09-14T02:18:00Z)
• Critical initialization destabilizes higher input derivatives in wide scalar-input networks [3.0318216701273397]
  We prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions.<n>For residual networks with branch scale L-1/2, we prove that every fixed finite derivative order has uniformly bounded variance.
  arXiv  Detail & Related papers  (2026-09-08T10:17:45Z)
• Solution-space heterogeneity shapes federated learning dynamics across partial differential equations [41.890949680958805]
  Federated machine learning enables institutions to train neural surrogates without centralizing local physical data.<n>Existing protocols partition coordinates, coefficients, boundary conditions, or geometries according to equation-specific rules.<n>Here, we introduce solution-space PDE-Dirichlet, a protocol that converts continuous supervised responses into reusable solution bins.
  arXiv  Detail & Related papers  (2026-09-04T11:23:29Z)
• Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Sampling [9.271408028342496]
  Constant-stepsize temporal-difference (TD) learning is attractive for policy evaluation, but inference from a single Markov trajectory must account for serial dependence and a stepsize-dependent stationary target.<n>We establish a functional central limit theorem whose covariance retains the multiplicative component induced by the random TD matrix and the stationary iterate error.<n>Experiments on FrozenLake and Garnet illustrate stationary-target coverage, RR target correction, and the finite-sample behavior of the horizon-indexed design.
  arXiv  Detail & Related papers  (2026-08-11T13:19:28Z)
• Discretization and Statistical Consistency of Functional Flow Matching [0.0]
  We prove strong $L2$ convergence of finite conditional velocity targets for every strongly consistent sequence of finite-rank reconstructions.<n>For learned flows, coupling directly to a population superposition path yields an end-to-end Wasserstein bound.
  arXiv  Detail & Related papers  (2026-08-05T07:05:47Z)
• Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks [0.0]
  Positive quadratic networks admit the lowrank representation f_U(x)=xtop UUtop x, where UinmathbbRdtimes r is identifiable only up to right multiplication.<n>We study how this quotient structure governs training dynamics, curvature, recovery, and bias.
  arXiv  Detail & Related papers  (2026-07-28T12:04:46Z)
• Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent [11.327284468812948]
  We show that negative-shifted gradient descent escapes a structural constraint.<n>The filter is smooth at the would-be pole and mixed-sign-capable.<n>A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.
  arXiv  Detail & Related papers  (2026-07-24T16:42:46Z)
• Optimization Geometrodynamics: Variational Reduction and Interaction Curvature [0.0]
  We develop geometrodynamics as a variational theory of this hidden determinant pushforward.<n>For affine pre-reductions, the induced interaction curvature is the negative-semidefinite operator $-G*H-1G$.<n>We prove a global analytic bundle with closed totally geodesic fibers and a unique analytic nearest-controller section.
  arXiv  Detail & Related papers  (2026-07-07T18:42:00Z)
• Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data [3.7437735326653883]
  This paper studies the rate-controlled middle case for full-batch linear classification on separable data.<n>For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice.
  arXiv  Detail & Related papers  (2026-07-07T08:55:35Z)
• Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse [16.78534503577613]
  We show that the input weights and biases align along finitely many directions, leading to an effective width collapse.<n>We establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data.
  arXiv  Detail & Related papers  (2026-07-03T22:03:13Z)
• Divergence is Uncertainty: A Closed-Form Posterior Covariance for Flow Matching [8.603039700922809]
  Flow matching has become a leading framework for generative modeling, but quantifying the uncertainty of its samples remains an open problem.<n>We show that none of these trade-offs is necessary.<n>Experiments on MNIST confirm that the resulting per-pixel uncertainty maps are semantically meaningful.
  arXiv  Detail & Related papers  (2026-05-01T04:25:00Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• Sampling and estimation on manifolds using the Langevin diffusion [45.57801520690309]
  Two estimators of linear functionals of $mu_phi $ based on the discretized Markov process are considered.<n>Error bounds are derived for sampling and estimation using a discretization of an intrinsically defined Langevin diffusion.
  arXiv  Detail & Related papers  (2023-12-22T18:01:11Z)
• Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterization [48.65416821017865]
  This chapter introduces a new algorithmic approach, dubbed scaled gradient (ScaledGD) It converges linearly at a constant rate independent of the condition number of the low-rank object. It maintains the low periteration cost of gradient descent for a variety of tasks.
  arXiv  Detail & Related papers  (2023-10-09T21:16:57Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction [49.66486092259376]
  The mean-field Langevin dynamics (MFLD) is a nonlinear generalization of the Langevin dynamics that incorporates a distribution-dependent drift. Recent works have shown that MFLD globally minimizes an entropy-regularized convex functional in the space of measures. We provide a framework to prove a uniform-in-time propagation of chaos for MFLD that takes into account the errors due to finite-particle approximation, time-discretization, and gradient approximation.
  arXiv  Detail & Related papers  (2023-06-12T16:28:11Z)
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability [69.01076284478151]
  In machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime.
  arXiv  Detail & Related papers  (2023-05-19T16:24:47Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.