Contrastive Learning Via Equivariant Representation

• URL: http://arxiv.org/abs/2406.00262v1
• Date: Sat, 1 Jun 2024 01:53:51 GMT
• Title: Contrastive Learning Via Equivariant Representation
• Authors: Sifan Song, Jinfeng Wang, Qiaochu Zhao, Xiang Li, Dufan Wu, Angelos Stefanidis, Jionglong Su, S. Kevin Zhou, Quanzheng Li,
• Abstract summary: We propose a novel Equivariant-based Contrastive Learning (ECL) framework, CLeVER (Contrastive Learning Via Equivariant Representation) Experimental results demonstrate that CLeVER effectively extracts and incorporates equivariant information from data, thereby improving the training efficiency and robustness of baseline models in downstream tasks.
• Score: 19.112460889771423
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Invariant-based Contrastive Learning (ICL) methods have achieved impressive performance across various domains. However, the absence of latent space representation for distortion (augmentation)-related information in the latent space makes ICL sub-optimal regarding training efficiency and robustness in downstream tasks. Recent studies suggest that introducing equivariance into Contrastive Learning (CL) can improve overall performance. In this paper, we rethink the roles of augmentation strategies and equivariance in improving CL efficacy. We propose a novel Equivariant-based Contrastive Learning (ECL) framework, CLeVER (Contrastive Learning Via Equivariant Representation), compatible with augmentation strategies of arbitrary complexity for various mainstream CL methods and model frameworks. Experimental results demonstrate that CLeVER effectively extracts and incorporates equivariant information from data, thereby improving the training efficiency and robustness of baseline models in downstream tasks.

Related papers

• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters. We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• FeTrIL++: Feature Translation for Exemplar-Free Class-Incremental Learning with Hill-Climbing [3.533544633664583]
  Exemplar-free class-incremental learning (EFCIL) poses significant challenges, primarily due to catastrophic forgetting. Traditional EFCIL approaches typically skew towards either model plasticity through successive fine-tuning or stability. This paper builds upon the foundational FeTrIL framework to examine the efficacy of various oversampling techniques and dynamic optimization strategies.
  arXiv  Detail & Related papers  (2024-03-12T08:34:05Z)
• On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning [71.44986275228747]
  In-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs) However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration)
  arXiv  Detail & Related papers  (2023-12-21T11:55:10Z)
• Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning [57.83232242068982]
  Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms. It remains unclear which attributes of DA account for its effectiveness in achieving sample-efficient visual RL. This work conducts comprehensive experiments to assess the impact of DA's attributes on its efficacy.
  arXiv  Detail & Related papers  (2023-05-25T15:46:20Z)
• Cross-Stream Contrastive Learning for Self-Supervised Skeleton-Based Action Recognition [22.067143671631303]
  Self-supervised skeleton-based action recognition enjoys a rapid growth along with the development of contrastive learning. We propose a Cross-Stream Contrastive Learning framework for skeleton-based action Representation learning (CSCLR) Specifically, the proposed CSCLR not only utilizes intra-stream contrast pairs, but introduces inter-stream contrast pairs as hard samples to formulate a better representation learning.
  arXiv  Detail & Related papers  (2023-05-03T10:31:35Z)
• ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations [30.745749133759304]
  We develop a theoretical framework to analyze the transferability of self-supervised contrastive learning. We show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL)
  arXiv  Detail & Related papers  (2023-03-02T09:26:20Z)
• Unbiased and Efficient Self-Supervised Incremental Contrastive Learning [31.763904668737304]
  We propose a self-supervised Incremental Contrastive Learning (ICL) framework consisting of a novel Incremental InfoNCE (NCE-II) loss function. ICL achieves up to 16.7x training speedup and 16.8x faster convergence with competitive results.
  arXiv  Detail & Related papers  (2023-01-28T06:11:31Z)
• Latent Variable Representation for Reinforcement Learning [131.03944557979725]
  It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning. We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
  arXiv  Detail & Related papers  (2022-12-17T00:26:31Z)
• Weak Augmentation Guided Relational Self-Supervised Learning [80.0680103295137]
  We introduce a novel relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Our proposed method employs sharpened distribution of pairwise similarities among different instances as textitrelation metric. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures.
  arXiv  Detail & Related papers  (2022-03-16T16:14:19Z)
• Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
  We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss. Experimental results show that our model defines a new state of the art for various datasets and settings.
  arXiv  Detail & Related papers  (2021-12-10T20:46:13Z)
• Invariance-based Multi-Clustering of Latent Space Embeddings for Equivariant Learning [12.770012299379099]
  We present an approach to disentangle equivariance feature maps in a Lie group manifold by enforcing deep, group-invariant learning. Our experiments show that this model effectively learns to disentangle the invariant and equivariant representations with significant improvements in the learning rate.
  arXiv  Detail & Related papers  (2021-07-25T03:27:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.