Related papers: Invariance-adapted decomposition and Lasso-type contrastive learning

Invariance-adapted decomposition and Lasso-type contrastive learning

URL: http://arxiv.org/abs/2210.07413v1
Date: Thu, 13 Oct 2022 23:30:12 GMT
Title: Invariance-adapted decomposition and Lasso-type contrastive learning
Authors: Masanori Koyama, Takeru Miyato, Kenji Fukumizu
Abstract summary: We show that contrastive learning is capable of decomposing the data space into the space that is invariant to all augmentations and its complement. This decomposition generalizes the one introduced in citetcontent_isolate and describes a structure that is analogous to the frequencies in the harmonic analysis of a group.
Score: 30.974508897223124
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Recent years have witnessed the effectiveness of contrastive learning in obtaining the representation of dataset that is useful in interpretation and downstream tasks. However, the mechanism that describes this effectiveness have not been thoroughly analyzed, and many studies have been conducted to investigate the data structures captured by contrastive learning. In particular, the recent study of \citet{content_isolate} has shown that contrastive learning is capable of decomposing the data space into the space that is invariant to all augmentations and its complement. In this paper, we introduce the notion of invariance-adapted latent space that decomposes the data space into the intersections of the invariant spaces of each augmentation and their complements. This decomposition generalizes the one introduced in \citet{content_isolate}, and describes a structure that is analogous to the frequencies in the harmonic analysis of a group. We experimentally show that contrastive learning with lasso-type metric can be used to find an invariance-adapted latent space, thereby suggesting a new potential for the contrastive learning. We also investigate when such a latent space can be identified up to mixings within each component.

Related papers

Ensemble Visualization With Variational Autoencoder [4.539358010738346]
We present a new method to visualize data ensembles by constructing structured probabilistic representations in latent spaces.<n>Preliminary results on a weather forecasting ensemble demonstrate the effectiveness and versatility of our method.
arXiv Detail & Related papers (2025-09-16T12:13:15Z)
The "Law" of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities [23.188014611990152]
This paper builds on work on the geometry and probabilistic interpretation of contrastive representations. We show how these representations can answer many of the same inferences as probabilistic graphical models. Our analysis suggests two new ways of using contrastive representations: in settings with pre-trained contrastive models, and for handling language ambiguity in reinforcement learning.
arXiv Detail & Related papers (2025-01-20T08:10:15Z)
Comparing the information content of probabilistic representation spaces [3.7277730514654555]
Probabilistic representation spaces convey information about a dataset, and to understand the effects of factors such as training loss and network architecture, we seek to compare the information content of such spaces. Here, instead of building upon point-based measures of comparison, we build upon classic methods from literature on hard clustering. We propose a practical method of estimation that is based on fingerprinting a representation space with a sample of the dataset and is applicable when the communicated information is only a handful of bits.
arXiv Detail & Related papers (2024-05-31T17:33:07Z)
Hodge-Aware Contrastive Learning [101.56637264703058]
Simplicial complexes prove effective in modeling data with multiway dependencies. We develop a contrastive self-supervised learning approach for processing simplicial data.
arXiv Detail & Related papers (2023-09-14T00:40:07Z)
On the Joint Interaction of Models, Data, and Features [82.60073661644435]
We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
arXiv Detail & Related papers (2023-06-07T21:35:26Z)
Decorr: Environment Partitioning for Invariant Learning and OOD Generalization [10.799855921851332]
Invariant learning methods are aimed at identifying a consistent predictor across multiple environments. When environments aren't inherent in the data, practitioners must define them manually. This environment partitioning affects invariant learning's efficacy but remains underdiscussed. In this paper, we suggest partitioning the dataset into several environments by isolating low-correlation data subsets.
arXiv Detail & Related papers (2022-11-18T06:49:35Z)
Correlation between Alignment-Uniformity and Performance of Dense Contrastive Representations [11.266613717084788]
We analyze the theoretical ideas of dense contrastive learning using a standard CNN and straightforward feature matching scheme. We discover the core principle in constructing a positive pair of dense features and empirically proved its validity. Also, we introduce a new scalar metric that summarizes the correlation between alignment-and-uniformity and downstream performance.
arXiv Detail & Related papers (2022-10-17T08:08:37Z)
Dynamic Latent Separation for Deep Learning [67.62190501599176]
A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
arXiv Detail & Related papers (2022-10-07T17:56:53Z)
Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
We propose a novel approach to identify meaningful and independent factors of variation in a dataset. Our method involves two separate latent subspaces for the target property and the remaining input information. We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
arXiv Detail & Related papers (2021-11-25T17:33:12Z)
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style [32.20957709045773]
We formulate the augmentation process as a latent variable model. We study the identifiability of the latent representation based on pairs of views of the observations. We introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies.
arXiv Detail & Related papers (2021-06-08T18:18:09Z)
Where and What? Examining Interpretable Disentangled Representations [96.32813624341833]
Capturing interpretable variations has long been one of the goals in disentanglement learning. Unlike the independence assumption, interpretability has rarely been exploited to encourage disentanglement in the unsupervised setting. In this paper, we examine the interpretability of disentangled representations by investigating two questions: where to be interpreted and what to be interpreted.
arXiv Detail & Related papers (2021-04-07T11:22:02Z)
Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
arXiv Detail & Related papers (2020-07-25T08:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.