MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck

• URL: http://arxiv.org/abs/2403.19078v1
• Date: Thu, 28 Mar 2024 00:50:02 GMT
• Title: MVEB: Self-Supervised Learning with Multi-View Entropy Bottleneck
• Authors: Liangjian Wen, Xiasi Wang, Jianzhuang Liu, Zenglin Xu,
• Abstract summary: Self-supervised approaches regard two views of an image as both the input and the self-supervised signals. Recent studies show that discarding superfluous information not shared between the views can improve generalization. We propose an objective multi-view entropy bottleneck (MVEB) to learn minimal sufficient representation effectively.
• Score: 53.44358636312935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning aims to learn representation that can be effectively generalized to downstream tasks. Many self-supervised approaches regard two views of an image as both the input and the self-supervised signals, assuming that either view contains the same task-relevant information and the shared information is (approximately) sufficient for predicting downstream tasks. Recent studies show that discarding superfluous information not shared between the views can improve generalization. Hence, the ideal representation is sufficient for downstream tasks and contains minimal superfluous information, termed minimal sufficient representation. One can learn this representation by maximizing the mutual information between the representation and the supervised view while eliminating superfluous information. Nevertheless, the computation of mutual information is notoriously intractable. In this work, we propose an objective termed multi-view entropy bottleneck (MVEB) to learn minimal sufficient representation effectively. MVEB simplifies the minimal sufficient learning to maximizing both the agreement between the embeddings of two views and the differential entropy of the embedding distribution. Our experiments confirm that MVEB significantly improves performance. For example, it achieves top-1 accuracy of 76.9\% on ImageNet with a vanilla ResNet-50 backbone on linear evaluation. To the best of our knowledge, this is the new state-of-the-art result with ResNet-50.

Related papers

• Rethinking Multi-view Representation Learning via Distilled Disentangling [34.14711778177439]
  Multi-view representation learning aims to derive robust representations that are both view-consistent and view-specific from diverse data sources. This paper presents an in-depth analysis of existing approaches in this domain, highlighting the redundancy between view-consistent and view-specific representations. We propose an innovative framework for multi-view representation learning, which incorporates a technique we term 'distilled disentangling'
  arXiv  Detail & Related papers  (2024-03-16T11:21:24Z)
• MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments [72.6405488990753]
  Self-supervised learning can be used for mitigating the greedy needs of Vision Transformer networks. We propose a single-stage and standalone method, MOCA, which unifies both desired properties. We achieve new state-of-the-art results on low-shot settings and strong experimental results in various evaluation protocols.
  arXiv  Detail & Related papers  (2023-07-18T15:46:20Z)
• Variational Distillation for Multi-View Learning [104.17551354374821]
  We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning. Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
  arXiv  Detail & Related papers  (2022-06-20T03:09:46Z)
• Provable Benefit of Multitask Representation Learning in Reinforcement Learning [46.11628795660159]
  This paper theoretically characterizes the benefit of representation learning under the low-rank Markov decision process (MDP) model. To the best of our knowledge, this is the first theoretical study that characterizes the benefit of representation learning in exploration-based reward-free multitask reinforcement learning.
  arXiv  Detail & Related papers  (2022-06-13T04:29:02Z)
• Rethinking Minimal Sufficient Representation in Contrastive Learning [28.83450836832452]
  We show that contrastive learning models have the risk of over-fitting to the shared information between views. We propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information. It significantly improves the performance of several classic contrastive learning models in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-14T11:17:48Z)
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments [57.33699905852397]
  We propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Our method simultaneously clusters the data while enforcing consistency between cluster assignments. Our method can be trained with large and small batches and can scale to unlimited amounts of data.
  arXiv  Detail & Related papers  (2020-06-17T14:00:42Z)
• What Makes for Good Views for Contrastive Learning? [90.49736973404046]
  We argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. We devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. As a by-product, we achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification.
  arXiv  Detail & Related papers  (2020-05-20T17:59:57Z)
• Evolving Losses for Unsupervised Video Representation Learning [91.2683362199263]
  We present a new method to learn video representations from large-scale unlabeled video data. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods.
  arXiv  Detail & Related papers  (2020-02-26T16:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.