Transformative or Conservative? Conservation laws for ResNets and Transformers

• URL: http://arxiv.org/abs/2506.06194v1
• Date: Fri, 06 Jun 2025 15:53:35 GMT
• Title: Transformative or Conservative? Conservation laws for ResNets and Transformers
• Abstract summary: This paper bridges the gap by deriving and analyzing conservation laws for modern architectures.<n>We first show that basic building blocks such as ReLU (or linear) shallow networks, with or without convolution, have easily expressed conservation laws.<n>We then introduce the notion of conservation laws that depend only on a subset of parameters.
• Score: 28.287184613608435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practical architectures. This paper bridges this gap by deriving and analyzing conservation laws for modern architectures, with a focus on convolutional ResNets and Transformer networks. For this, we first show that basic building blocks such as ReLU (or linear) shallow networks, with or without convolution, have easily expressed conservation laws, and no more than the known ones. In the case of a single attention layer, we also completely describe all conservation laws, and we show that residual blocks have the same conservation laws as the same block without a skip connection. We then introduce the notion of conservation laws that depend only on a subset of parameters (corresponding e.g. to a pair of consecutive layers, to a residual block, or to an attention layer). We demonstrate that the characterization of such laws can be reduced to the analysis of the corresponding building block in isolation. Finally, we examine how these newly discovered conservation principles, initially established in the continuous gradient flow regime, persist under discrete optimization dynamics, particularly in the context of Stochastic Gradient Descent (SGD).

Related papers

• Basin-Preserving Discretizations of Modern Hopfield Retrieval Dynamics: Energy Cells, Dissipation, and the Attention Limit [0.0]
  We show that every finite energy cell below the escape energy is contained simultaneously in the basins the continuous flow, attracting every relaxed attention.<n>We also derive explicit local bounds, with certified optimal tunneling preservation patterns.
  arXiv  Detail & Related papers  (2026-08-21T17:06:43Z)
• CoFINN: Conservation Flux Informed Neural Networks for Physics Problems Governed by Conservation Laws [0.46976113832881716]
  CoFINN is a physics-informed deep learning framework for predicting compressible flow fields governed by conservation laws.<n>CoFINN embeds finite-volume conservation physics directly into the training process.<n>Results show that CoFINN improves aerodynamic force prediction accuracy, reducing drag prediction error by up to 34% at extreme angles of attack.
  arXiv  Detail & Related papers  (2026-07-05T15:21:12Z)
• Conservation Laws for Modern Neural Architectures [3.344515796046363]
  This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs.
  arXiv  Detail & Related papers  (2026-06-16T11:44:53Z)
• FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction [51.56484100374058]
  Streaming 3D reconstruction maintains a persistent latent state that is updated online from incoming frames.<n>FILT3R casts recurrent state updates as state estimation in token space.<n>Code will be released at https://github.com/jinotter3/FILT3R.
  arXiv  Detail & Related papers  (2026-03-19T04:56:36Z)
• Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks [52.153950303594684]
  We propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF)<n>AGF accurately captures the network's structural "kinetic utility"<n>We design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors.
  arXiv  Detail & Related papers  (2026-03-12T18:19:21Z)
• Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA [50.494504099850325]
  We introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear.<n>We show this constraint improves signal-to-noise ratio, and preserves diversity by preventing collisions during trajectory.<n>We demonstrate that geometric priors can surpass brute-force scaling.
  arXiv  Detail & Related papers  (2026-02-26T04:45:07Z)
• RecTok: Reconstruction Distillation along Rectified Flow [85.51292475005151]
  We propose RecTok, which overcomes the limitations of high-dimensional visual tokenizers through two key innovations.<n>Our method distills the semantic information in VFMs into the forward flow trajectories in flow matching.<n>Our RecTok achieves superior image reconstruction, generation quality, and discriminative performance.
  arXiv  Detail & Related papers  (2025-12-15T15:14:20Z)
• Geometric-Disentangelment Unlearning [106.99160454669902]
  gradient ascent on forget samples often harms retained knowledge.<n>We propose the Geometric-disment Unlearning (GU) that decomposes any candidate forget gradient update into tangential and normal components to retain space and executes only the normal component.<n>Our method is plug-and-play and can be attached to existing gradient-based unlearning procedures to mitigate side effects.
  arXiv  Detail & Related papers  (2025-11-21T09:58:25Z)
• An Exterior-Embedding Neural Operator Framework for Preserving Conservation Laws [11.631493226237042]
  We propose Exterior-Embedded Conservation Framework (ECF), a universal conserving framework that can be integrated with various data-driven neural operators to enforce conservation laws strictly in predictions.<n>ECF consists of two key components: a conservation quantity encoder that extracts conserved quantities from input data, and a conservation quantity decoder that adjusts the neural operator's predictions using these quantities to ensure strict conservation compliance in the final output.
  arXiv  Detail & Related papers  (2025-11-20T17:29:19Z)
• Adaptive Initial Residual Connections for GNNs with Theoretical Guarantees [7.831509538890674]
  We investigate an adaptive residual scheme in which different nodes have varying residual strengths.<n>We prove that this approach prevents oversmoothing; particularly, we show that the Dirichlet energy of the embeddings remains bounded away from zero.<n>This is the first theoretical guarantee not only for the adaptive setting, but also for static residual connections with activation functions.
  arXiv  Detail & Related papers  (2025-11-10T01:08:37Z)
• Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning [73.18052192964349]
  We develop a theoretical framework that explains how discrete symbolic structures can emerge naturally from continuous neural network training dynamics.<n>By lifting neural parameters to a measure space and modeling training as Wasserstein gradient flow, we show that under geometric constraints, the parameter measure $mu_t$ undergoes two concurrent phenomena.
  arXiv  Detail & Related papers  (2025-06-26T22:40:30Z)
• A Signed Graph Approach to Understanding and Mitigating Oversmoothing in GNNs [54.62268052283014]
  We present a unified theoretical perspective based on the framework of signed graphs.<n>We show that many existing strategies implicitly introduce negative edges that alter message-passing to resist oversmoothing.<n>We propose Structural Balanced Propagation (SBP), a plug-and-play method that assigns signed edges based on either labels or feature similarity.
  arXiv  Detail & Related papers  (2025-02-17T03:25:36Z)
• Dynamical freezing in the thermodynamic limit: the strongly driven ensemble [37.31317754926534]
  A periodically driven (Floquet) system in the absence of any conservation law heats to a featureless infinite temperature' state. Here, we find--for a clean and interacting generic spin chain--that this can be prevented by the emergence of it approximate but stable conservation-laws not present in the undriven system. We show numerically, it in the thermodynamic limit,' that when required by these emergent conservation-laws, the entanglement-entropy density of an infinite subsystem remains zero.
  arXiv  Detail & Related papers  (2024-10-14T19:57:43Z)
• Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows [28.287184613608435]
  We show that conservation laws for momentum-based dynamics exhibit temporal dependence. We also observe a "conservation loss" when transitioning from gradient flow to momentum dynamics. This phenomenon also manifests in non-Euclidean metrics, used e.g. for Nonnegative Matrix Factorization (NMF)
  arXiv  Detail & Related papers  (2024-05-21T15:59:55Z)
• Simple Cycle Reservoirs are Universal [0.358439716487063]
  Reservoir models form a subclass of recurrent neural networks with fixed non-trainable input and dynamic coupling weights. We show that they are capable of universal approximation of any unrestricted linear reservoir system.
  arXiv  Detail & Related papers  (2023-08-21T15:35:59Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)
• Towards Practical Control of Singular Values of Convolutional Layers [65.25070864775793]
  Convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties. We offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity.
  arXiv  Detail & Related papers  (2022-11-24T19:09:44Z)
• Log-linear Guardedness and its Implications [116.87322784046926]
  Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful. This work formally defines the notion of log-linear guardedness as the inability of an adversary to predict the concept directly from the representation. We show that, in the binary case, under certain assumptions, a downstream log-linear model cannot recover the erased concept.
  arXiv  Detail & Related papers  (2022-10-18T17:30:02Z)
• Accumulative reservoir construction: Bridging continuously relaxed and periodically refreshed extended reservoirs [0.0]
  We introduce an accumulative reservoir construction that employs a series of partial refreshes of the extended reservoirs. This provides a unified framework for both continuous (Lindblad) relaxation and a recently introduced periodically refresh approach. We show how the range of behavior impacts errors and the computational cost, including within tensor networks.
  arXiv  Detail & Related papers  (2022-10-10T17:59:58Z)
• Orthogonalizing Convolutional Layers with the Cayley Transform [83.73855414030646]
  We propose and evaluate an alternative approach to parameterize convolutional layers that are constrained to be orthogonal. We show that our method indeed preserves orthogonality to a high degree even for large convolutions.
  arXiv  Detail & Related papers  (2021-04-14T23:54:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.