Conservation Laws for Modern Neural Architectures

• URL: http://arxiv.org/abs/2606.17816v1
• Date: Tue, 16 Jun 2026 11:44:53 GMT
• Title: Conservation Laws for Modern Neural Architectures
• Abstract summary: This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs.
• Score: 3.344515796046363
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.

Related papers

• Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime [24.053364183688874]
  Training dynamics is central to understanding neural networks.<n>We propose a convergence framework for analyzing gradient descent dynamics under a broad family of neural network architectures.
  arXiv  Detail & Related papers  (2026-06-22T14:00:26Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• A Structure-Preserving Graph Neural Solver for Parametric Hyperbolic Conservation Laws [1.988145627448243]
  We present an interpretable, structure-preserving graph neural solver that bridges classical numerical principles with graph neural networks (GNNs)<n>Inspired by Arbitrary high-order DERivatives schemes, we further recast message-passing GNNs as high-order space-time predictors.<n>The neural solver achieves superior long-horizon rollout stability and accuracy compared with strong surrogate baselines.
  arXiv  Detail & Related papers  (2026-04-17T01:45:54Z)
• Catastrophic Forgetting in Kolmogorov-Arnold Networks [27.683054983159835]
  Catastrophic forgetting is a longstanding challenge in continual learning.<n>Recent architectural advances like Kolmogorov-Arnold Networks (KANs) have been suggested to offer intrinsic resistance to forgetting.<n>We present a comprehensive study of catastrophic forgetting in KANs and develop a theoretical framework that links forgetting to activation support overlap and intrinsic data dimension.
  arXiv  Detail & Related papers  (2025-11-16T23:22:50Z)
• Information-Theoretic Framework for Understanding Modern Machine-Learning [4.435094091999926]
  We present an information-theoretic framework that views learning as universal prediction under log loss.<n>We argue that successful architectures possess a broad complexity range, enabling learning in highly over- parameterized model classes.<n>The framework sheds light on the role of inductive biases, the effectiveness of descent gradient, and phenomena such as flat minima.
  arXiv  Detail & Related papers  (2025-06-09T11:32:31Z)
• Transformative or Conservative? Conservation laws for ResNets and Transformers [28.287184613608435]
  This paper bridges the gap by deriving and analyzing conservation laws for modern architectures.<n>We first show that basic building blocks such as ReLU (or linear) shallow networks, with or without convolution, have easily expressed conservation laws.<n>We then introduce the notion of conservation laws that depend only on a subset of parameters.
  arXiv  Detail & Related papers  (2025-06-06T15:53:35Z)
• Conservation-informed Graph Learning for Spatiotemporal Dynamics Prediction [84.26340606752763]
  In this paper, we introduce the conservation-informed GNN (CiGNN), an end-to-end explainable learning framework.<n>The network is designed to conform to the general symmetry conservation law via symmetry where conservative and non-conservative information passes over a multiscale space by a latent temporal marching strategy.<n>Results demonstrate that CiGNN exhibits remarkable baseline accuracy and generalizability, and is readily applicable to learning for prediction of varioustemporal dynamics.
  arXiv  Detail & Related papers  (2024-12-30T13:55:59Z)
• State-space models can learn in-context by gradient descent [1.3087858009942543]
  We show that state-space models can perform gradient-based learning and use it for in-context learning in much the same way as transformers.<n>Specifically, we prove that a single structured state-space model layer, augmented with multiplicative input and output gating, can reproduce the outputs of an implicit linear model.<n>We also provide novel insights into the relationship between state-space models and linear self-attention, and their ability to learn in-context.
  arXiv  Detail & Related papers  (2024-10-15T15:22:38Z)
• From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks [68.8204255655161]
  We propose a new type of continuous-time control system, called AutoencODE, based on a controlled field that drives dynamics. We show that many architectures can be recovered in regions where the loss function is locally convex.
  arXiv  Detail & Related papers  (2023-07-05T13:26:17Z)
• An Ode to an ODE [78.97367880223254]
  We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d) This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
  arXiv  Detail & Related papers  (2020-06-19T22:05:19Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.