Probing Equivariance and Symmetry Breaking in Convolutional Networks

• URL: http://arxiv.org/abs/2501.01999v3
• Date: Mon, 02 Jun 2025 02:43:00 GMT
• Title: Probing Equivariance and Symmetry Breaking in Convolutional Networks
• Authors: Sharvaree Vadgama, Mohammad Mohaiminul Islam, Domas Buracas, Christian Shewmake, Artem Moskalev, Erik Bekkers,
• Abstract summary: We introduce textttRapidash, a unified group convolutional architecture that allows for different variants of equivariant and non-equivariant models.<n>Our results suggest that more constrained equivariant models outperform less constrained alternatives when aligned with the geometry of the task.
• Score: 3.241907749876342
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we explore the trade-offs of explicit structural priors, particularly group equivariance. We address this through theoretical analysis and a comprehensive empirical study. To enable controlled and fair comparisons, we introduce \texttt{Rapidash}, a unified group convolutional architecture that allows for different variants of equivariant and non-equivariant models. Our results suggest that more constrained equivariant models outperform less constrained alternatives when aligned with the geometry of the task, and increasing representation capacity does not fully eliminate performance gaps. We see improved performance of models with equivariance and symmetry-breaking through tasks like segmentation, regression, and generation across diverse datasets. Explicit \textit{symmetry breaking} via geometric reference frames consistently improves performance, while \textit{breaking equivariance} through geometric input features can be helpful when aligned with task geometry. Our results provide task-specific performance trends that offer a more nuanced way for model selection.

Related papers

• Parameter-free approximate equivariance for tasks with finite group symmetry [15.964726158869777]
  Equivariant neural networks incorporate symmetries through group actions, embedding them as an inductive bias to improve performance on a wide variety of tasks.<n>We propose a simple zero- parameter approach that imposes approximate equivariance for a finite group in the latent representation, as an additional term in the loss function.<n>We benchmark our approach on three datasets and compare it against several existing equivariant methods, showing that it achieves similar or better performance for a fraction of the parameters.
  arXiv  Detail & Related papers  (2025-06-09T21:23:26Z)
• Learning (Approximately) Equivariant Networks via Constrained Optimization [25.51476313302483]
  Equivariant neural networks are designed to respect symmetries through their architecture.<n>Real-world data often departs from perfect symmetry because of noise, structural variation, measurement bias, or other symmetry-breaking effects.<n>We introduce Adaptive Constrained Equivariance (ACE), a constrained optimization approach that starts with a flexible, non-equivariant model.
  arXiv  Detail & Related papers  (2025-05-19T18:08:09Z)
• EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networks [55.214444066134114]
  We design a fully target-equivariant architecture-ensuring permutation invariance via equivariant encoders, decoders, and a bi-attention mechanism.<n> Empirical evaluation on standard classification benchmarks shows that, on datasets with more classes than those seen during pre-training, our model matches or surpasses existing methods while incurring lower computational overhead.
  arXiv  Detail & Related papers  (2025-02-10T17:11:20Z)
• Approximate Equivariance in Reinforcement Learning [35.04248486334824]
  We develop approximately equivariant algorithms in reinforcement learning. Results show that the approximately equivariant network performs on par with exactly equivariant networks when exact symmetries are present.
  arXiv  Detail & Related papers  (2024-11-06T19:44:46Z)
• Does equivariance matter at scale? [15.247352029530523]
  We study how equivariant and non-equivariant networks scale with compute and training samples. First, equivariance improves data efficiency, but training non-equivariant models with data augmentation can close this gap given sufficient epochs. Second, scaling with compute follows a power law, with equivariant models outperforming non-equivariant ones at each tested compute budget.
  arXiv  Detail & Related papers  (2024-10-30T16:36:59Z)
• Relaxed Equivariance via Multitask Learning [7.905957228045955]
  We introduce REMUL, a training procedure for approximating equivariance with multitask learning. We show that unconstrained models can learn approximate symmetries by minimizing an additional simple equivariance loss. Our method achieves competitive performance compared to equivariant baselines while being $10 times$ faster at inference and $2.5 times$ at training.
  arXiv  Detail & Related papers  (2024-10-23T13:50:27Z)
• Improving Equivariant Model Training via Constraint Relaxation [31.507956579770088]
  We propose a novel framework for improving the optimization of such models by relaxing the hard equivariance constraint during training.<n>We provide experimental results on different state-of-the-art network architectures, demonstrating how this training framework can result in equivariant models with improved generalization performance.
  arXiv  Detail & Related papers  (2024-08-23T17:35:08Z)
• Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance [16.49488981364657]
  We present a novel framework to overcome the limitations of equivariant architectures in learning functions with group symmetries. We use an arbitrary base model such as anvariant or a transformer and symmetrize it to be equivariant to the given group. Empirical tests show competitive results against tailored equivariant architectures.
  arXiv  Detail & Related papers  (2023-06-05T13:40:54Z)
• Deep Neural Networks with Efficient Guaranteed Invariances [77.99182201815763]
  We address the problem of improving the performance and in particular the sample complexity of deep neural networks. Group-equivariant convolutions are a popular approach to obtain equivariant representations. We propose a multi-stream architecture, where each stream is invariant to a different transformation.
  arXiv  Detail & Related papers  (2023-03-02T20:44:45Z)
• Self-supervised learning of Split Invariant Equivariant representations [0.0]
  We introduce 3DIEBench, consisting of renderings from 3D models over 55 classes and more than 2.5 million images where we have full control on the transformations applied to the objects. We introduce a predictor architecture based on hypernetworks to learn equivariant representations with no possible collapse to invariance. We introduce SIE (Split Invariant-Equivariant) which combines the hypernetwork-based predictor with representations split in two parts, one invariant, the other equivariant, to learn richer representations.
  arXiv  Detail & Related papers  (2023-02-14T07:53:18Z)
• Architectural Optimization over Subgroups for Equivariant Neural Networks [0.0]
  We propose equivariance relaxation morphism and $[G]$-mixed equivariant layer to operate with equivariance constraints on a subgroup. We present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization.
  arXiv  Detail & Related papers  (2022-10-11T14:37:29Z)
• The Lie Derivative for Measuring Learned Equivariance [84.29366874540217]
  We study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. We find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities. For example, transformers can be more equivariant than convolutional neural networks after training.
  arXiv  Detail & Related papers  (2022-10-06T15:20:55Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks [79.13089902898848]
  Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. We show that different tasks in computer vision require features to encode different (in)variances.
  arXiv  Detail & Related papers  (2021-11-22T18:16:35Z)
• Commutative Lie Group VAE for Disentanglement Learning [96.32813624341833]
  We view disentanglement learning as discovering an underlying structure that equivariantly reflects the factorized variations shown in data. A simple model named Commutative Lie Group VAE is introduced to realize the group-based disentanglement learning. Experiments show that our model can effectively learn disentangled representations without supervision, and can achieve state-of-the-art performance without extra constraints.
  arXiv  Detail & Related papers  (2021-06-07T07:03:14Z)
• Learning Invariances in Neural Networks [51.20867785006147]
  We show how to parameterize a distribution over augmentations and optimize the training loss simultaneously with respect to the network parameters and augmentation parameters. We can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations.
  arXiv  Detail & Related papers  (2020-10-22T17:18:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.