A Canonicalization Perspective on Invariant and Equivariant Learning

• URL: http://arxiv.org/abs/2405.18378v4
• Date: Mon, 04 Nov 2024 01:21:32 GMT
• Title: A Canonicalization Perspective on Invariant and Equivariant Learning
• Authors: George Ma, Yifei Wang, Derek Lim, Stefanie Jegelka, Yisen Wang,
• Abstract summary: We introduce a canonicalization perspective that provides an essential and complete view of the design of frames. We show that there exists an inherent connection between frames and canonical forms. We design novel frames for eigenvectors that are strictly superior to existing methods.
• Score: 54.44572887716977
• License:
• Abstract: In many applications, we desire neural networks to exhibit invariance or equivariance to certain groups due to symmetries inherent in the data. Recently, frame-averaging methods emerged to be a unified framework for attaining symmetries efficiently by averaging over input-dependent subsets of the group, i.e., frames. What we currently lack is a principled understanding of the design of frames. In this work, we introduce a canonicalization perspective that provides an essential and complete view of the design of frames. Canonicalization is a classic approach for attaining invariance by mapping inputs to their canonical forms. We show that there exists an inherent connection between frames and canonical forms. Leveraging this connection, we can efficiently compare the complexity of frames as well as determine the optimality of certain frames. Guided by this principle, we design novel frames for eigenvectors that are strictly superior to existing methods -- some are even optimal -- both theoretically and empirically. The reduction to the canonicalization perspective further uncovers equivalences between previous methods. These observations suggest that canonicalization provides a fundamental understanding of existing frame-averaging methods and unifies existing equivariant and invariant learning methods. Code is available at https://github.com/PKU-ML/canonicalization.

Related papers

• Metric Convolutions: A Unifying Theory to Adaptive Convolutions [3.481985817302898]
  Metric convolutions replace standard convolutions in image processing and deep learning. They require fewer parameters and provide better generalisation. Our approach shows competitive performance in standard denoising and classification tasks.
  arXiv  Detail & Related papers  (2024-06-08T08:41:12Z)
• Improved Canonicalization for Model Agnostic Equivariance [6.783232060611113]
  Building equivariant models using traditional methods requires designing equivariant versions of existing models and training them from scratch. We propose a new optimization-based method that employs any non-equivariant network for canonicalization. Our method uses contrastive learning to efficiently learn a canonical orientation and offers more flexibility for the choice of canonicalization network.
  arXiv  Detail & Related papers  (2024-05-23T01:34:12Z)
• Equivariant Frames and the Impossibility of Continuous Canonicalization [10.02508080274145]
  We show that unweighted frame-averaging can turn a smooth, non-symmetric function into a discontinuous, symmetric function. We construct efficient and continuous weighted frames for the actions of $SO(2)$, $SO(3)$, and $S_n$ on point clouds.
  arXiv  Detail & Related papers  (2024-02-25T12:40:42Z)
• Equivariance with Learned Canonicalization Functions [77.32483958400282]
  We show that learning a small neural network to perform canonicalization is better than using predefineds. Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
  arXiv  Detail & Related papers  (2022-11-11T21:58:15Z)
• Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation [88.14365009076907]
  Iterative refinement is a useful paradigm for representation learning. We develop an implicit differentiation approach that improves the stability and tractability of training.
  arXiv  Detail & Related papers  (2022-07-02T10:00:35Z)
• Frame Averaging for Equivariant Shape Space Learning [85.42901997467754]
  A natural way to incorporate symmetries in shape space learning is to ask that the mapping to the shape space (encoder) and mapping from the shape space (decoder) are equivariant to the relevant symmetries. We present a framework for incorporating equivariance in encoders and decoders by introducing two contributions.
  arXiv  Detail & Related papers  (2021-12-03T06:41:19Z)
• Frame Averaging for Invariant and Equivariant Network Design [50.87023773850824]
  We introduce Frame Averaging (FA), a framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types. We show that FA-based models have maximal expressive power in a broad setting. We propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs.
  arXiv  Detail & Related papers  (2021-10-07T11:05:23Z)
• Bayesian Attention Modules [65.52970388117923]
  We propose a scalable version of attention that is easy to implement and optimize. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.
  arXiv  Detail & Related papers  (2020-10-20T20:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.