Subhomogeneous Deep Equilibrium Models

• URL: http://arxiv.org/abs/2403.00720v2
• Date: Thu, 6 Jun 2024 17:59:38 GMT
• Title: Subhomogeneous Deep Equilibrium Models
• Authors: Pietro Sittoni, Francesco Tudisco,
• Abstract summary: Implicit-depth neural networks have grown as powerful alternatives to traditional networks in various applications. These models often lack guarantees of existence and uniqueness, raising stability, performance, and issues. We present a new analysis of the existence and uniqueness of fixed points for implicit-depth neural networks based on the concept of subhomogeneous operators and the nonlinear Perron-Frobenius theory.
• Score: 5.1334640820688175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Implicit-depth neural networks have grown as powerful alternatives to traditional networks in various applications in recent years. However, these models often lack guarantees of existence and uniqueness, raising stability, performance, and reproducibility issues. In this paper, we present a new analysis of the existence and uniqueness of fixed points for implicit-depth neural networks based on the concept of subhomogeneous operators and the nonlinear Perron-Frobenius theory. Compared to previous similar analyses, our theory allows for weaker assumptions on the parameter matrices, thus yielding a more flexible framework for well-defined implicit networks. We illustrate the performance of the resulting subhomogeneous networks on feedforward, convolutional, and graph neural network examples.

Related papers

• On the Stability of the Jacobian Matrix in Deep Neural Networks [9.617753464544606]
  We establish a general stability theorem for deep neural networks that accommodates sparsity and weakly correlated weights.<n>Our results rely on recent advances in random matrix theory, and provide rigorous guarantees for spectral stability in a much broader class of network models.
  arXiv  Detail & Related papers  (2025-06-10T13:04:42Z)
• A Non-negative VAE:the Generalized Gamma Belief Network [49.970917207211556]
  The gamma belief network (GBN) has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. We introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. We also propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables.
  arXiv  Detail & Related papers  (2024-08-06T18:18:37Z)
• Enhancing lattice kinetic schemes for fluid dynamics with Lattice-Equivariant Neural Networks [79.16635054977068]
  We present a new class of equivariant neural networks, dubbed Lattice-Equivariant Neural Networks (LENNs) Our approach develops within a recently introduced framework aimed at learning neural network-based surrogate models Lattice Boltzmann collision operators. Our work opens towards practical utilization of machine learning-augmented Lattice Boltzmann CFD in real-world simulations.
  arXiv  Detail & Related papers  (2024-05-22T17:23:15Z)
• Sparsity-aware generalization theory for deep neural networks [12.525959293825318]
  We present a new approach to analyzing generalization for deep feed-forward ReLU networks. We show fundamental trade-offs between sparsity and generalization.
  arXiv  Detail & Related papers  (2023-07-01T20:59:05Z)
• A New PHO-rmula for Improved Performance of Semi-Structured Networks [0.0]
  We show that techniques to properly identify the contributions of the different model components in SSNs lead to suboptimal network estimation. We propose a non-invasive post-hocization (PHO) that guarantees identifiability of model components and provides better estimation and prediction quality. Our theoretical findings are supported by numerical experiments, a benchmark comparison as well as a real-world application to COVID-19 infections.
  arXiv  Detail & Related papers  (2023-06-01T10:23:28Z)
• Generalization and Estimation Error Bounds for Model-based Neural Networks [78.88759757988761]
  We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks. We derive practical design rules that allow to construct model-based networks with guaranteed high generalization.
  arXiv  Detail & Related papers  (2023-04-19T16:39:44Z)
• Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth [92.25666446274188]
  sinusoidal neural networks with activations have been proposed as an alternative to networks with traditional activation functions. We first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth.
  arXiv  Detail & Related papers  (2022-11-26T07:41:48Z)
• The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks [26.58848653965855]
  We introduce the class of quasi-homogeneous models, which is expressive enough to describe nearly all neural networks with homogeneous activations. We find that gradient flow implicitly favors a subset of the parameters, unlike in the case of a homogeneous model where all parameters are treated equally.
  arXiv  Detail & Related papers  (2022-10-07T21:14:09Z)
• Provably Training Neural Network Classifiers under Fairness Constraints [70.64045590577318]
  We show that overparametrized neural networks could meet the constraints. Key ingredient of building a fair neural network classifier is establishing no-regret analysis for neural networks.
  arXiv  Detail & Related papers  (2020-12-30T18:46:50Z)
• DebiNet: Debiasing Linear Models with Nonlinear Overparameterized Neural Networks [11.04121146441257]
  We incorporate over- parameterized neural networks into semi-parametric models to bridge the gap between inference and prediction. We show the theoretical foundations that make this possible and demonstrate with numerical experiments. We propose a framework, DebiNet, in which we plug-in arbitrary feature selection methods to our semi-parametric neural network.
  arXiv  Detail & Related papers  (2020-11-01T04:12:53Z)
• Monotone operator equilibrium networks [97.86610752856987]
  We develop a new class of implicit-depth model based on the theory of monotone operators, the Monotone Operator Equilibrium Network (monDEQ) We show the close connection between finding the equilibrium point of an implicit network and solving a form of monotone operator splitting problem. We then develop a parameterization of the network which ensures that all operators remain monotone, which guarantees the existence of a unique equilibrium point.
  arXiv  Detail & Related papers  (2020-06-15T17:57:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.