Activation Saturation and Floquet Spectrum Collapse in Neural ODEs

• URL: http://arxiv.org/abs/2604.00543v1
• Date: Wed, 01 Apr 2026 06:39:56 GMT
• Title: Activation Saturation and Floquet Spectrum Collapse in Neural ODEs
• Abstract summary: This is a collapse of the Floquet spectrum: as saturation deepens, all exponents are driven to zero.<n>As a secondary contribution, a saturation- spectral factorisation yields a refined bound $widetildeC(U)le C(U)$ whose improvement is exponentially amplified in$T$ at the flow level.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs $\dot{h}=f_θ(h)$ with saturating activations ($\tanh$, sigmoid, etc.): if $q$ hidden layers of the MLP $f_θ$ satisfy $|σ'|\leδ$ on a region~$U$, the input Jacobian is attenuated as $\norm{Df_θ(x)}\le C(U)$ (for activations with $\sup_{x}|σ'(x)|\le 1$, e.g.\ $\tanh$ and sigmoid, this reduces to $C_Wδ^q$), forcing every Floquet (Lyapunov) exponen along any $T$-periodic orbit $γ\subset U$ into the interval $[-C(U),\;C(U)]$. This is a collapse of the Floquet spectrum: as saturation deepens ($δ\to 0$), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural -- it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with $σ'>0$, a saturation-weighted spectral factorisation yields a refined bound $\widetilde{C}(U)\le C(U)$ whose improvement is amplified exponentially in~$T$ at the flow level. All results are numerically illustrated on the Stuart--Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of $\tanh$-NODEs on the Morris--Lecar neuron model.

Related papers

• The Information Content of Krylov Observables: A Machine Learning Approach [0.0]
  We quantify the spread complexity $mathcalC(t)$, the discrete Wigner negativity $N(t)$, and the normalized negativity $(t)=N(t)/|S(t)|$, with $S(t)$ the survival amplitude, recently proposed as a second-moment infall probe (arXiv:2607.04065).<n>Small residual networks (16-32 neurons) and boosted trees are trained on half of $sim 57,000$ labeled evolutions spanning the GUE, GOE and Poisson ensembles, the integrable $SL
  arXiv  Detail & Related papers  (2026-07-19T17:17:21Z)
• Sequential Physics-Constrained Neural Operator Forward Modeling for the $\ extit{Norne}$ Reservoir System [3.175986877079391]
  We develop a comprehensive framework for sequential surrogate modeling of black-oil reservoir dynamics using neural operators.<n>The application focus is the Norne benchmark reservoir, defined on a heterogeneous $46times112times22$ grid.<n>A 1000-member ensemble runs in under one minute on a single B200 GPU, giving a $sim104times$ wall-clock speedup over the OPM finite-volume simulator.
  arXiv  Detail & Related papers  (2026-05-27T17:18:39Z)
• Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent [0.0]
  We prove the Spectral Alignment Decomposition: $= dlog_k / dlog_k$, where $_k$ measures alignment between Kronecker factor eigenbases and gradient singular directions.<n>This reduces "why does $$ vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads.
  arXiv  Detail & Related papers  (2026-05-22T23:31:38Z)
• The Spectral Edge Thesis: A Mathematical Framework for Intra-Signal Phase Transitions in Neural Network Training [0.0]
  We show that phase transitions in neural network training are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates.<n>Adiabatic parameter $mathcalA = |G|_F / (, g2)$ controls circuit stability: $mathcalA ll 1$ (plateau), $mathcalA sim 1$ (phase transition), $mathcalA gg 1$ (forgetting)
  arXiv  Detail & Related papers  (2026-03-30T20:10:22Z)
• Rényi exponent landscape of multipartite entanglement in free-fermion systems [51.56484100374058]
  We show that the Rényi tripartite information $I_3() exhibits a qualitatively $exclusion-dependent scaling at small Fermi momentum.<n>A direct consequence is a replica obstruction: $I_m(n)/I_m(1) sim zm-1 to 0$ for all integer $n geq 2$, so the leading von Neumann signal cannot be reconstructed from integer Rényi data.
  arXiv  Detail & Related papers  (2026-03-09T22:27:00Z)
• Information-Computation Tradeoffs for Noiseless Linear Regression with Oblivious Contamination [65.37519531362157]
  We show that any efficient Statistical Query algorithm for this task requires VSTAT complexity at least $tildeOmega(d1/2/alpha2)$.
  arXiv  Detail & Related papers  (2025-10-12T15:42:44Z)
• Emergence and scaling laws in SGD learning of shallow neural networks [64.48316762675141]
  We study the complexity of online gradient descent (SGD) for learning a two-layer neural network with $P$ neurons on isotropic Gaussian data.<n>We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective.
  arXiv  Detail & Related papers  (2025-04-28T16:58:55Z)
• Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit [75.4661041626338]
  We study the problem of gradient descent learning of a single-index target function $f_*(boldsymbolx) = textstylesigma_*left(langleboldsymbolx,boldsymbolthetarangleright)$<n>We prove that a two-layer neural network optimized by an SGD-based algorithm learns $f_*$ with a complexity that is not governed by information exponents.
  arXiv  Detail & Related papers  (2024-06-03T17:56:58Z)
• Interplay between depth and width for interpolation in neural ODEs [0.0]
  We examine the interplay between their width $p$ and number of layer transitions $L$. In the high-dimensional setting, we demonstrate that $p=O(N)$ neurons are likely sufficient to achieve exact control.
  arXiv  Detail & Related papers  (2024-01-18T11:32:50Z)
• A Unified Framework for Uniform Signal Recovery in Nonlinear Generative Compressed Sensing [68.80803866919123]
  Under nonlinear measurements, most prior results are non-uniform, i.e., they hold with high probability for a fixed $mathbfx*$ rather than for all $mathbfx*$ simultaneously. Our framework accommodates GCS with 1-bit/uniformly quantized observations and single index models as canonical examples. We also develop a concentration inequality that produces tighter bounds for product processes whose index sets have low metric entropy.
  arXiv  Detail & Related papers  (2023-09-25T17:54:19Z)
• Learning a Single Neuron with Adversarial Label Noise via Gradient Descent [50.659479930171585]
  We study a function of the form $mathbfxmapstosigma(mathbfwcdotmathbfx)$ for monotone activations. The goal of the learner is to output a hypothesis vector $mathbfw$ that $F(mathbbw)=C, epsilon$ with high probability.
  arXiv  Detail & Related papers  (2022-06-17T17:55:43Z)
• Superharmonic double-well systems with zero-energy ground states: Relevance for diffusive relaxation scenarios [0.0]
  Relaxation properties of the Smoluchowski diffusion process on a line can be spectrally quantified. A peculiarity of $hatH$ is that it refers to a family of quasi-exactly solvable Schr"odinger-type systems.
  arXiv  Detail & Related papers  (2021-04-24T08:11:54Z)
• Agnostic Learning of a Single Neuron with Gradient Descent [92.7662890047311]
  We consider the problem of learning the best-fitting single neuron as measured by the expected square loss. For the ReLU activation, our population risk guarantee is $O(mathsfOPT1/2)+epsilon$. For the ReLU activation, our population risk guarantee is $O(mathsfOPT1/2)+epsilon$.
  arXiv  Detail & Related papers  (2020-05-29T07:20:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.