Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensors

• URL: http://arxiv.org/abs/2412.04594v2
• Date: Tue, 14 Jan 2025 11:03:05 GMT
• Title: Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensors
• Authors: Putri A. van der Linden, Alejandro García-Castellanos, Sharvaree Vadgama, Thijs P. Kuipers, Erik J. Bekkers,
• Abstract summary: Group equivariance has emerged as a valuable inductive bias in deep learning. Group equivariant methods require the groups of interest to be known beforehand. We show that when the dataset exhibits strong symmetries, the permutation matrices will converge to regular group representations.
• Score: 46.59269589647962
• License:
• Abstract: Group equivariance has emerged as a valuable inductive bias in deep learning, enhancing generalization, data efficiency, and robustness. Classically, group equivariant methods require the groups of interest to be known beforehand, which may not be realistic for real-world data. Additionally, baking in fixed group equivariance may impose overly restrictive constraints on model architecture. This highlights the need for methods that can dynamically discover and apply symmetries as soft constraints. For neural network architectures, equivariance is commonly achieved through group transformations of a canonical weight tensor, resulting in weight sharing over a given group $G$. In this work, we propose to learn such a weight-sharing scheme by defining a collection of learnable doubly stochastic matrices that act as soft permutation matrices on canonical weight tensors, which can take regular group representations as a special case. This yields learnable kernel transformations that are jointly optimized with downstream tasks. We show that when the dataset exhibits strong symmetries, the permutation matrices will converge to regular group representations and our weight-sharing networks effectively become regular group convolutions. Additionally, the flexibility of the method enables it to effectively pick up on partial symmetries.

Related papers

• A Diagrammatic Approach to Improve Computational Efficiency in Group Equivariant Neural Networks [1.9643748953805935]
  Group equivariant neural networks are growing in importance owing to their ability to generalise well in applications where the data has known underlying symmetries. Recent characterisations of a class of these networks that use high-order tensor power spaces as their layers suggest that they have significant potential. We present a fast matrix multiplication algorithm for any equivariant weight matrix that maps between tensor power layer spaces in these networks for four groups.
  arXiv  Detail & Related papers  (2024-12-14T14:08:06Z)
• Symmetry Discovery for Different Data Types [52.2614860099811]
  Equivariant neural networks incorporate symmetries into their architecture, achieving higher generalization performance. We propose LieSD, a method for discovering symmetries via trained neural networks which approximate the input-output mappings of the tasks. We validate the performance of LieSD on tasks with symmetries such as the two-body problem, the moment of inertia matrix prediction, and top quark tagging.
  arXiv  Detail & Related papers  (2024-10-13T13:39:39Z)
• Lie Algebra Canonicalization: Equivariant Neural Operators under arbitrary Lie Groups [11.572188414440436]
  We propose Lie aLgebrA Canonicalization (LieLAC), a novel approach that exploits only the action of infinitesimal generators of the symmetry group. operating within the framework of canonicalization, LieLAC can easily be integrated with unconstrained pre-trained models.
  arXiv  Detail & Related papers  (2024-10-03T17:21:30Z)
• Metric Learning for Clifford Group Equivariant Neural Networks [15.551447911164903]
  Clifford Group Equivariant Neural Networks (CGENNs) leverage Clifford algebras and multivectors to ensure symmetry constraints in neural representations. Previous works have restricted internal network representations to Euclidean or Minkowski (pseudo-)metrics, handpicked depending on the problem at hand. We propose an alternative method that enables the metric to be learned in a data-driven fashion, allowing the CGENN network to learn more flexible representations.
  arXiv  Detail & Related papers  (2024-07-13T15:41:14Z)
• Lie Group Decompositions for Equivariant Neural Networks [12.139222986297261]
  We show how convolution kernels can be parametrized to build models equivariant with respect to affine transformations. We evaluate the robustness and out-of-distribution generalisation capability of our model on the benchmark affine-invariant classification task.
  arXiv  Detail & Related papers  (2023-10-17T16:04:33Z)
• Accelerated Discovery of Machine-Learned Symmetries: Deriving the Exceptional Lie Groups G2, F4 and E6 [55.41644538483948]
  This letter introduces two improved algorithms that significantly speed up the discovery of symmetry transformations. Given the significant complexity of the exceptional Lie groups, our results demonstrate that this machine-learning method for discovering symmetries is completely general and can be applied to a wide variety of labeled datasets.
  arXiv  Detail & Related papers  (2023-07-10T20:25:44Z)
• Generative Adversarial Symmetry Discovery [19.098785309131458]
  LieGAN represents symmetry as interpretable Lie algebra basis and can discover various symmetries. The learned symmetry can also be readily used in several existing equivariant neural networks to improve accuracy and generalization in prediction.
  arXiv  Detail & Related papers  (2023-02-01T04:28:36Z)
• Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principles [55.41644538483948]
  We design a deep-learning algorithm for the discovery and identification of the continuous group of symmetries present in a labeled dataset. We use fully connected neural networks to model the transformations symmetry and the corresponding generators. Our study also opens the door for using a machine learning approach in the mathematical study of Lie groups and their properties.
  arXiv  Detail & Related papers  (2023-01-13T16:25:25Z)
• Learning Graphical Factor Models with Riemannian Optimization [70.13748170371889]
  This paper proposes a flexible algorithmic framework for graph learning under low-rank structural constraints. The problem is expressed as penalized maximum likelihood estimation of an elliptical distribution. We leverage geometries of positive definite matrices and positive semi-definite matrices of fixed rank that are well suited to elliptical models.
  arXiv  Detail & Related papers  (2022-10-21T13:19:45Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.