Response Renormalization for Critical Deep Equilibrium Models

• URL: http://arxiv.org/abs/2608.23725v1
• Date: Mon, 24 Aug 2026 18:10:30 GMT
• Title: Response Renormalization for Critical Deep Equilibrium Models
• Abstract summary: Training through implicit differentiation requires solving an adjoint system built from the residual Jacobian.<n>We introduce Response Renormalization, a backward-pass framework that lifts selected near-pole denominators while leaving unlifted response channels unchanged.<n>We derive dense and matrix-free collective formulations, distinguish exact gradients of a modified frozen-anchor residual from backward-response surrogates, and extend the construction to Structured Implicit Layers and Vector Attractors.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Equilibrium Models (DEQs) compute predictions from a hidden representation unchanged by the model update. Training through this equilibrium uses implicit differentiation and requires solving an adjoint system built from the residual Jacobian. If this Jacobian is nearly singular along loss-sensitive directions, small perturbations can be strongly amplified in the adjoint response, producing large, highly sensitive gradients that can make optimization unreliable. We introduce Response Renormalization, a backward-pass framework that lifts selected near-pole denominators while leaving unlifted response channels unchanged. Collective Mode Response Renormalization (CMR) applies this correction in a low-dimensional critical subspace, while Phi-adaptive CMR computes a bounded response mass from a positive susceptibility rule. We derive dense and matrix-free collective formulations, distinguish exact gradients of a modified frozen-anchor residual from backward-response surrogates, and extend the construction to Structured Implicit Layers and Vector Attractors (SILVA). Across 23 multiphysics families spanning partial differential equations, three-dimensional fields, operator maps, complex geometries, and particle systems, CMR and Phi-CMR yield test errors no more than five percent higher than those from models trained with exact implicit differentiation in more than 98% of static and 95% of transient family-seed comparisons. Solver-index experiments show convergence toward the static adjoint, while physical-time rollouts retain predictive fidelity under the evaluated conditions. These results demonstrate that selective response renormalization can control near-critical adjoint amplification without globally damping well-conditioned sensitivity. Therefore, the method can make parameter updates more reliable while preserving the useful gradient information needed for learning.

Related papers

• Phase Transition in Binary Compressed Sensing via Annealing with Adaptive Regularization [0.7734726150561088]
  Regularization choice changes the recovery phase diagrams of binary compressed sensing.<n>We develop a regularization-selection method that combines systematic parameter search with random forest regression.
  arXiv  Detail & Related papers  (2026-09-15T06:35:58Z)
• Sensitivity-Constrained Neural Operators for Data-Efficient Forward and Inverse Modeling of Partial Differential Equation Systems [7.72071408793585]
  We study sensitivity-constrained neural operators (SC-NOs)<n>SC-NOs augment standard training with solver-derived Jacobian supervision.<n>We evaluate SC-NOs on advection--diffusion and RANS--Spalart--Allmaras benchmarks, input-dimensionality scaling tests, long-horizon autoregressive rollout, and a shallow-water Tohoku tsunami source-inversion case.
  arXiv  Detail & Related papers  (2026-08-30T16:37:25Z)
• Effective Covariance Dynamics in Solvable High-Dimensional GANs [2.639628765667673]
  We study a solvable high-dimensional model of generative adversarial network (GAN) training in which a linear generator learns a low-dimensional subspace from data.<n>We prove that the microscopic training process converges, in the high-dimensional limit, to deterministic ordinary differential equations governed by this effective covariance.
  arXiv  Detail & Related papers  (2026-06-25T16:33:54Z)
• Coverage Analysis of Rydberg Atom Quantum Receiver Arrays: A Stochastic Geometry Approach [55.47164192369501]
  Rydberg atomic quantum receivers (RAQRs) offer quantum-limited sensitivity and broadband tunability.<n>In dense deployments, aggregate interference can push atomic transducer out of its small-signal regime.<n>This paper addresses the question by embedding the RAQR front end into a geometry coverage analysis.
  arXiv  Detail & Related papers  (2026-05-22T04:05:13Z)
• Next-Acceleration-Scale Prediction for Autoregressive MRI Reconstruction [52.32112533846212]
  MRI reconstruction is an inherently ill-posed inverse problem, since incomplete measurements admit many plausible solutions.<n>We address this limitation by moving reconstruction to discrete multi-scale latent space and posing it as autoregressive next-acceleration-scale prediction.<n>We show that our approach delivers improved reconstruction performance across diverse sampling patterns under extreme undersampling.
  arXiv  Detail & Related papers  (2026-05-19T04:40:50Z)
• EquiReg: Equivariance Regularized Diffusion for Inverse Problems [67.01847869495558]
  We propose EquiReg diffusion, a framework for regularizing posterior sampling in diffusion-based inverse problem solvers.<n>When applied to a variety of solvers, EquiReg outperforms state-of-the-art diffusion models in both linear and nonlinear image restoration tasks.
  arXiv  Detail & Related papers  (2025-05-29T01:25:43Z)
• Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion [55.95767828747407]
  In domains such as molecular and protein generation, physical systems exhibit inherent symmetries that are critical to model.<n>We present a framework that reduces training variance and provides a provably lower-variance gradient estimator.<n>We also present a practical implementation of this estimator incorporating the loss and sampling procedure through a method we call Orbit Diffusion.
  arXiv  Detail & Related papers  (2025-02-14T03:26:57Z)
• Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias [8.668428992331808]
  We develop an Sobolev norm learning curve for kernel ridge(less) regression when addressing (elliptical) linear inverse problems.<n>Our results show that the PDE operators in the inverse problem can stabilize the variance and even behave benign overfitting for fixed-dimensional problems.
  arXiv  Detail & Related papers  (2024-06-13T14:54:30Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems [64.29491112653905]
  We propose a novel and efficient diffusion sampling strategy that synergistically combines the diffusion sampling and Krylov subspace methods. Specifically, we prove that if tangent space at a denoised sample by Tweedie's formula forms a Krylov subspace, then the CG with the denoised data ensures the data consistency update to remain in the tangent space. Our proposed method achieves more than 80 times faster inference time than the previous state-of-the-art method.
  arXiv  Detail & Related papers  (2023-03-10T07:42:49Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.