Homomorphic Self-Supervised Learning

• URL: http://arxiv.org/abs/2211.08282v1
• Date: Tue, 15 Nov 2022 16:32:36 GMT
• Title: Homomorphic Self-Supervised Learning
• Authors: T. Anderson Keller, Xavier Suau, Luca Zappella
• Abstract summary: We introduce a general framework we call Homomorphic Self-Supervised Learning. We show how it may subsume the use of input-augmentations provided an augmentation-homomorphic feature extractor.
• Score: 1.0742675209112622
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we observe that many existing self-supervised learning algorithms can be both unified and generalized when seen through the lens of equivariant representations. Specifically, we introduce a general framework we call Homomorphic Self-Supervised Learning, and theoretically show how it may subsume the use of input-augmentations provided an augmentation-homomorphic feature extractor. We validate this theory experimentally for simple augmentations, demonstrate how the framework fails when representational structure is removed, and further empirically explore how the parameters of this framework relate to those of traditional augmentation-based self-supervised learning. We conclude with a discussion of the potential benefits afforded by this new perspective on self-supervised learning.

Related papers

• Toward Understanding In-context vs. In-weight Learning [50.24035812301655]
  We identify simplified distributional properties that give rise to the emergence and disappearance of in-context learning. We then extend the study to a full large language model, showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.
  arXiv  Detail & Related papers  (2024-10-30T14:09:00Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Flow Factorized Representation Learning [109.51947536586677]
  We introduce a generative model which specifies a distinct set of latent probability paths that define different input transformations. We show that our model achieves higher likelihoods on standard representation learning benchmarks while simultaneously being closer to approximately equivariant models.
  arXiv  Detail & Related papers  (2023-09-22T20:15:37Z)
• On the Joint Interaction of Models, Data, and Features [82.60073661644435]
  We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
  arXiv  Detail & Related papers  (2023-06-07T21:35:26Z)
• Homomorphism Autoencoder -- Learning Group Structured Representations from Observed Transitions [51.71245032890532]
  We propose methods enabling an agent acting upon the world to learn internal representations of sensory information consistent with actions that modify it. In contrast to existing work, our approach does not require prior knowledge of the group and does not restrict the set of actions the agent can perform.
  arXiv  Detail & Related papers  (2022-07-25T11:22:48Z)
• RELAX: Representation Learning Explainability [10.831313203043514]
  We propose RELAX, which is the first approach for attribution-based explanations of representations. ReLAX explains representations by measuring similarities in the representation space between an input and masked out versions of itself. We provide theoretical interpretations of RELAX and conduct a novel analysis of feature extractors trained using supervised and unsupervised learning.
  arXiv  Detail & Related papers  (2021-12-19T14:51:31Z)
• Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning [79.4957965474334]
  Key goal of unsupervised representation learning is "inverting" a data generating process to recover its latent properties. This paper asks, "Can we instead identify latent properties by leveraging knowledge of the mechanisms that govern their evolution?" We provide a complete characterization of the sources of non-identifiability as we vary knowledge about a set of possible mechanisms.
  arXiv  Detail & Related papers  (2021-10-29T14:04:08Z)
• Interpreting and improving deep-learning models with reality checks [13.287382944078562]
  This chapter covers recent work aiming to interpret models by attributing importance to features and feature groups for a single prediction. We show how these attributions can be used to directly improve the generalization of a neural network or to distill it into a simple model.
  arXiv  Detail & Related papers  (2021-08-16T00:58:15Z)
• Leveraging Hidden Structure in Self-Supervised Learning [2.385916960125935]
  We propose a principled framework based on a mutual information objective, which integrates self-supervised and structure learning. Preliminary experiments on CIFAR-10 show that the proposed framework achieves higher generalization performance in downstream classification tasks.
  arXiv  Detail & Related papers  (2021-06-30T13:35:36Z)
• A Self-Supervised Framework for Function Learning and Extrapolation [1.9374999427973014]
  We present a framework for how a learner may acquire representations that support generalization. We show the resulting representations outperform those from other models for unsupervised time series learning.
  arXiv  Detail & Related papers  (2021-06-14T12:41:03Z)
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective [27.93916012334704]
  Unsupervised and self-supervised learning approaches have become a crucial tool to learn representations for downstream prediction tasks. We present a unifying perspective where several such approaches can be viewed as imposing a regularization on the representation via a learnable function using unlabeled data. We propose a discriminative theoretical framework for analyzing the sample complexity of these approaches, which generalizes the framework of (Balcan and Blum, 2010) to allow learnable regularization functions.
  arXiv  Detail & Related papers  (2020-08-06T04:06:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.