Soft Equivariance Regularization for Invariant Self-Supervised Learning

• URL: http://arxiv.org/abs/2603.06693v1
• Date: Wed, 04 Mar 2026 13:36:17 GMT
• Title: Soft Equivariance Regularization for Invariant Self-Supervised Learning
• Abstract summary: Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations.<n>We propose Soft Equivariance Regularization (SER), a plug-in regularizer that decouples where invariance and equivariance are enforced.<n>SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only 1.008x training FLOPs.
• Score: 23.047550451521662
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations. While effective for recognition, enforcing strong invariance can suppress transformation-dependent structure that is useful for robustness to geometric perturbations and spatially sensitive transfer. A growing body of work, therefore, augments invariance-based SSL with equivariance objectives, but these objectives are often imposed on the same final representation. We empirically observe a trade-off in this coupled setting: pushing equivariance regularization toward deeper layers improves equivariance scores but degrades ImageNet-1k linear evaluation, motivating a layer-decoupled design. Motivated by this trade-off, we propose Soft Equivariance Regularization (SER), a plug-in regularizer that decouples where invariance and equivariance are enforced: we keep the base SSL objective unchanged on the final embedding, while softly encouraging equivariance on an intermediate spatial token map via analytically specified group actions $ρ_g$ applied directly in feature space. SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only 1.008x training FLOPs. On ImageNet-1k ViT-S/16 pretraining, SER improves MoCo-v3 by +0.84 Top-1 in linear evaluation under a strictly matched 2-view setting and consistently improves DINO and Barlow Twins; under matched view counts, SER achieves the best ImageNet-1k linear-eval Top-1 among the compared invariance+equivariance add-ons. SER further improves ImageNet-C/P by +1.11/+1.22 Top-1 and frozen-backbone COCO detection by +1.7 mAP. Finally, applying the same layer-decoupling recipe to existing invariance+equivariance baselinesimproves their accuracy, suggesting layer decoupling as a general design principle for combining invariance and equivariance.

Related papers

• VISReg: Variance-Invariance-Sketching Regularization for JEPA training [21.894781659564817]
  Self-supervised learning methods prevent embedding collapse via modelings or explicit regularization of the embedding space.<n>We propose Variance-Invariance-Sketching Regularization (VISReg)<n>VISReg replaces covariance with a Sliced-Wasserstein-based sketching objective that enforces full distributional shape.<n>We show that VISReg scales linearly, outperforms existing regularization on low-quality datasets, and is resilient to long-tailed and low-rank regimes.
  arXiv  Detail & Related papers  (2026-06-01T17:58:01Z)
• When Does Margin Clamping Affect Training Variance? Dataset-Dependent Effects in Contrastive Forward-Forward Learning [0.0]
  Contrastive Forward-Forward (CFF) learning trains Vision Transformers layer by layer against supervised contrastive objectives.<n>We focus on one implementation detail: the positive-pair margin in the contrastive loss is applied through saturating similarity clamping.<n>We prove that an alternative formulation, subtracting the margin after the log-probability, is gradient-neutral under the mean-over-positives reduction.
  arXiv  Detail & Related papers  (2026-03-01T07:00:38Z)
• SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representations [1.2431372110088466]
  We introduce SEIS, a subspace metric for analyzing layer-wise feature representations under geometric transformations.<n>Synthetic validation confirms that SEIS correctly recovers known transformations.<n>We show that multi-task learning induces synergistic gains in both properties at the shared encoder.
  arXiv  Detail & Related papers  (2026-02-03T22:38:08Z)
• Information Hidden in Gradients of Regression with Target Noise [2.8911861322232686]
  We show that the gradients alone can reveal the Hessian.<n>We provide non-asymptotic operator-norm guarantees under sub-Gaussian inputs.
  arXiv  Detail & Related papers  (2026-01-26T14:50:16Z)
• Truly Scale-Equivariant Deep Nets with Fourier Layers [14.072558848402362]
  In computer vision, models must be able to adapt to changes in image resolution to effectively carry out tasks such as image segmentation. Recent works have made progress in developing scale-equivariant convolutional neural networks, through weight-sharing and kernel resizing. We propose a novel architecture based on Fourier layers to achieve truly scale-equivariant deep nets.
  arXiv  Detail & Related papers  (2023-11-06T07:32:27Z)
• Learning to Transform for Generalizable Instance-wise Invariance [48.647925994707855]
  Given any image, we use a normalizing flow to predict a distribution over transformations and average the predictions over them. This normalizing flow is trained end-to-end and can learn a much larger range of transformations than Augerino and InstaAug. When used as data augmentation, our method shows accuracy and robustness gains on CIFAR 10, CIFAR10-LT, and TinyImageNet.
  arXiv  Detail & Related papers  (2023-09-28T17:59:58Z)
• Equivariant Similarity for Vision-Language Foundation Models [134.77524524140168]
  This study focuses on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks. We propose EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs. Compared to the existing evaluation sets, EqBen is the first to focus on "visual-minimal change"
  arXiv  Detail & Related papers  (2023-03-25T13:22:56Z)
• Soft Augmentation for Image Classification [68.71067594724663]
  We propose generalizing augmentation with invariant transforms to soft augmentation. We show that soft targets allow for more aggressive data augmentation. We also show that soft augmentations generalize to self-supervised classification tasks.
  arXiv  Detail & Related papers  (2022-11-09T01:04:06Z)
• Regularising for invariance to data augmentation improves supervised learning [82.85692486314949]
  We show that using multiple augmentations per input can improve generalisation. We propose an explicit regulariser that encourages this invariance on the level of individual model predictions.
  arXiv  Detail & Related papers  (2022-03-07T11:25:45Z)
• Improving the Sample-Complexity of Deep Classification Networks with Invariant Integration [77.99182201815763]
  Leveraging prior knowledge on intraclass variance due to transformations is a powerful method to improve the sample complexity of deep neural networks. We propose a novel monomial selection algorithm based on pruning methods to allow an application to more complex problems. We demonstrate the improved sample complexity on the Rotated-MNIST, SVHN and CIFAR-10 datasets.
  arXiv  Detail & Related papers  (2022-02-08T16:16:11Z)
• Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
  We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss. Experimental results show that our model defines a new state of the art for various datasets and settings.
  arXiv  Detail & Related papers  (2021-12-10T20:46:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.