Optimization Dynamics of Equivariant and Augmented Neural Networks

• URL: http://arxiv.org/abs/2303.13458v2
• Date: Thu, 21 Sep 2023 18:18:44 GMT
• Title: Optimization Dynamics of Equivariant and Augmented Neural Networks
• Authors: Axel Flinth and Fredrik Ohlsson
• Abstract summary: We investigate the optimization of multilayer perceptrons on symmetric data. We show that, under natural assumptions on the loss and non-linearities, the sets of equivariant stationary points are identical for the two strategies.
• Score: 3.706821920681978
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We investigate the optimization of multilayer perceptrons on symmetric data. We compare the strategy of constraining the architecture to be equivariant to that of using augmentation. We show that, under natural assumptions on the loss and non-linearities, the sets of equivariant stationary points are identical for the two strategies, and that the set of equivariant layers is invariant under the gradient flow for augmented models. Finally, we show that stationary points may be unstable for augmented training although they are stable for the equivariant models.

Related papers

• Symmetry Discovery for Different Data Types [52.2614860099811]
  Equivariant neural networks incorporate symmetries into their architecture, achieving higher generalization performance. We propose LieSD, a method for discovering symmetries via trained neural networks which approximate the input-output mappings of the tasks. We validate the performance of LieSD on tasks with symmetries such as the two-body problem, the moment of inertia matrix prediction, and top quark tagging.
  arXiv  Detail & Related papers  (2024-10-13T13:39:39Z)
• Equivariant score-based generative models provably learn distributions with symmetries efficiently [7.90752151686317]
  Empirical studies have demonstrated that incorporating symmetries into generative models can provide better generalization and sampling efficiency. We provide the first theoretical analysis and guarantees of score-based generative models (SGMs) for learning distributions that are invariant with respect to some group symmetry.
  arXiv  Detail & Related papers  (2024-10-02T05:14:28Z)
• Relative Representations: Topological and Geometric Perspectives [53.88896255693922]
  Relative representations are an established approach to zero-shot model stitching. We introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes.
  arXiv  Detail & Related papers  (2024-09-17T08:09:22Z)
• Enhancing lattice kinetic schemes for fluid dynamics with Lattice-Equivariant Neural Networks [79.16635054977068]
  We present a new class of equivariant neural networks, dubbed Lattice-Equivariant Neural Networks (LENNs) Our approach develops within a recently introduced framework aimed at learning neural network-based surrogate models Lattice Boltzmann collision operators. Our work opens towards practical utilization of machine learning-augmented Lattice Boltzmann CFD in real-world simulations.
  arXiv  Detail & Related papers  (2024-05-22T17:23:15Z)
• A Characterization Theorem for Equivariant Networks with Point-wise Activations [13.00676132572457]
  We prove that rotation-equivariant networks can only be invariant, as it happens for any network which is equivariant with respect to connected compact groups. We show that feature spaces of disentangled steerable convolutional neural networks are trivial representations.
  arXiv  Detail & Related papers  (2024-01-17T14:30:46Z)
• Learning Layer-wise Equivariances Automatically using Gradients [66.81218780702125]
  Convolutions encode equivariance symmetries into neural networks leading to better generalisation performance. symmetries provide fixed hard constraints on the functions a network can represent, need to be specified in advance, and can not be adapted. Our goal is to allow flexible symmetry constraints that can automatically be learned from data using gradients.
  arXiv  Detail & Related papers  (2023-10-09T20:22:43Z)
• Using and Abusing Equivariance [10.70891251559827]
  We show how Group Equivariant Convolutional Neural Networks use subsampling to learn to break equivariance to their symmetries. We show that a change in the input dimension of a network as small as a single pixel can be enough for commonly used architectures to become approximately equivariant, rather than exactly.
  arXiv  Detail & Related papers  (2023-08-22T09:49:26Z)
• Architectural Optimization over Subgroups for Equivariant Neural Networks [0.0]
  We propose equivariance relaxation morphism and $[G]$-mixed equivariant layer to operate with equivariance constraints on a subgroup. We present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization.
  arXiv  Detail & Related papers  (2022-10-11T14:37:29Z)
• Learning Symmetric Embeddings for Equivariant World Models [9.781637768189158]
  We propose learning symmetric embedding networks (SENs) that encode an input space (e.g. images) This network can be trained end-to-end with an equivariant task network to learn an explicitly symmetric representation. Our experiments demonstrate that SENs facilitate the application of equivariant networks to data with complex symmetry representations.
  arXiv  Detail & Related papers  (2022-04-24T22:31:52Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent [80.73580820014242]
  We focus on the problem of efficient sampling and learning of probability densities by incorporating symmetries in probabilistic models. We first introduce Equivariant Stein Variational Gradient Descent algorithm -- an equivariant sampling method based on Stein's identity for sampling from densities with symmetries. We propose new ways of improving and scaling up training of energy based models.
  arXiv  Detail & Related papers  (2021-06-15T01:35:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.