Self-Supervised Learning based on Transformed Image Reconstruction for Equivariance-Coherent Feature Representation

• URL: http://arxiv.org/abs/2503.18753v1
• Date: Mon, 24 Mar 2025 15:01:50 GMT
• Title: Self-Supervised Learning based on Transformed Image Reconstruction for Equivariance-Coherent Feature Representation
• Authors: Qin Wang, Benjamin Bruns, Hanno Scharr, Kai Krajsek,
• Abstract summary: We propose a self-supervised learning approach to learning computer vision features.<n>The system learns transformations independently by reconstructing images that have undergone previously unseen transformations.<n>Our approach performs strong on a rich set of realistic computer vision downstream tasks, almost always improving over all baselines.
• Score: 3.7622885602373626
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The equivariant behaviour of features is essential in many computer vision tasks, yet popular self-supervised learning (SSL) methods tend to constrain equivariance by design. We propose a self-supervised learning approach where the system learns transformations independently by reconstructing images that have undergone previously unseen transformations. Specifically, the model is tasked to reconstruct intermediate transformed images, e.g. translated or rotated images, without prior knowledge of these transformations. This auxiliary task encourages the model to develop equivariance-coherent features without relying on predefined transformation rules. To this end, we apply transformations to the input image, generating an image pair, and then split the extracted features into two sets per image. One set is used with a usual SSL loss encouraging invariance, the other with our loss based on the auxiliary task to reconstruct the intermediate transformed images. Our loss and the SSL loss are linearly combined with weighted terms. Evaluating on synthetic tasks with natural images, our proposed method strongly outperforms all competitors, regardless of whether they are designed to learn equivariance. Furthermore, when trained alongside augmentation-based methods as the invariance tasks, such as iBOT or DINOv2, we successfully learn a balanced combination of invariant and equivariant features. Our approach performs strong on a rich set of realistic computer vision downstream tasks, almost always improving over all baselines.

Related papers

• Self-supervised Transformation Learning for Equivariant Representations [26.207358743969277]
  Unsupervised representation learning has significantly advanced various machine learning tasks.<n>We propose Self-supervised Transformation Learning (STL), replacing transformation labels with transformation representations derived from image pairs.<n>We demonstrate the approach's effectiveness across diverse classification and detection tasks, outperforming existing methods in 7 out of 11 benchmarks.
  arXiv  Detail & Related papers  (2025-01-15T10:54:21Z)
• Equivariant Representation Learning for Augmentation-based Self-Supervised Learning via Image Reconstruction [3.7003845808210594]
  We propose integrating an image reconstruction task as an auxiliary component in augmentation-based self-supervised learning algorithms.<n>Our method implements a cross-attention mechanism to blend features learned from two augmented views, subsequently reconstructing one of them.<n>Results consistently demonstrate significant improvements over standard augmentation-based self-supervised learning methods.
  arXiv  Detail & Related papers  (2024-12-04T13:47:37Z)
• Self-Supervised Learning for Group Equivariant Neural Networks [75.62232699377877]
  Group equivariant neural networks are the models whose structure is restricted to commute with the transformations on the input. We propose two concepts for self-supervised tasks: equivariant pretext labels and invariant contrastive loss. Experiments on standard image recognition benchmarks demonstrate that the equivariant neural networks exploit the proposed self-supervised tasks.
  arXiv  Detail & Related papers  (2023-03-08T08:11:26Z)
• ParGAN: Learning Real Parametrizable Transformations [50.51405390150066]
  We propose ParGAN, a generalization of the cycle-consistent GAN framework to learn image transformations. The proposed generator takes as input both an image and a parametrization of the transformation. We show how, with disjoint image domains with no annotated parametrization, our framework can create smooths as well as learn multiple transformations simultaneously.
  arXiv  Detail & Related papers  (2022-11-09T16:16:06Z)
• Imaging with Equivariant Deep Learning [9.333799633608345]
  We review the emerging field of equivariant imaging and show how it can provide improved generalization and new imaging opportunities. We show the interplay between the acquisition physics and group actions and links to iterative reconstruction, blind compressed sensing and self-supervised learning.
  arXiv  Detail & Related papers  (2022-09-05T02:13:57Z)
• Equivariant Contrastive Learning [20.369942206674576]
  In state-of-the-art self-supervised learning (SSL) pre-training produces semantically good representations. We extend popular SSL methods to a more general framework which we name Equivariant Self-Supervised Learning (E-SSL) We demonstrate E-SSL's effectiveness empirically on several popular computer vision benchmarks.
  arXiv  Detail & Related papers  (2021-10-28T17:21:33Z)
• Robust Training Using Natural Transformation [19.455666609149567]
  We present NaTra, an adversarial training scheme to improve robustness of image classification algorithms. We target attributes of the input images that are independent of the class identification, and manipulate those attributes to mimic real-world natural transformations. We demonstrate the efficacy of our scheme by utilizing the disentangled latent representations derived from well-trained GANs.
  arXiv  Detail & Related papers  (2021-05-10T01:56:03Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Encoding Robustness to Image Style via Adversarial Feature Perturbations [72.81911076841408]
  We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce robust models. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training.
  arXiv  Detail & Related papers  (2020-09-18T17:52:34Z)
• FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning [64.32306537419498]
  We propose a novel learned feature-based refinement and augmentation method that produces a varied set of complex transformations. These transformations also use information from both within-class and across-class representations that we extract through clustering. We demonstrate that our method is comparable to current state of art for smaller datasets while being able to scale up to larger datasets.
  arXiv  Detail & Related papers  (2020-07-16T17:55:31Z)
• On Compositions of Transformations in Contrastive Self-Supervised Learning [66.15514035861048]
  In this paper, we generalize contrastive learning to a wider set of transformations. We find that being invariant to certain transformations and distinctive to others is critical to learning effective video representations.
  arXiv  Detail & Related papers  (2020-03-09T17:56:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.