Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations

• URL: http://arxiv.org/abs/2311.08815v2
• Date: Tue, 20 Aug 2024 15:33:12 GMT
• Title: Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations
• Authors: Cian Eastwood, Julius von Kügelgen, Linus Ericsson, Diane Bouchacourt, Pascal Vincent, Bernhard Schölkopf, Mark Ibrahim,
• Abstract summary: Self-supervised representation learning often uses data augmentations to induce "style" attributes of the data. It is difficult to deduce a priori which attributes of the data are indeed "style" and can be safely discarded. We introduce a more principled approach that seeks to disentangle style features rather than discard them.
• Score: 63.73044203154743
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised representation learning often uses data augmentations to induce some invariance to "style" attributes of the data. However, with downstream tasks generally unknown at training time, it is difficult to deduce a priori which attributes of the data are indeed "style" and can be safely discarded. To deal with this, current approaches try to retain some style information by tuning the degree of invariance to some particular task, such as ImageNet object classification. However, prior work has shown that such task-specific tuning can lead to significant performance degradation on other tasks that rely on the discarded style. To address this, we introduce a more principled approach that seeks to disentangle style features rather than discard them. The key idea is to add multiple style embedding spaces where: (i) each is invariant to all-but-one augmentation; and (ii) joint entropy is maximized. We formalize our structured data-augmentation procedure from a causal latent-variable-model perspective, and prove identifiability of both content and individual style variables. We empirically demonstrate the benefits of our approach on both synthetic and real-world data.

Related papers

• FUNCK: Information Funnels and Bottlenecks for Invariant Representation Learning [7.804994311050265]
  We investigate a set of related information funnels and bottleneck problems that claim to learn invariant representations from the data. We propose a new element to this family of information-theoretic objectives: The Conditional Privacy Funnel with Side Information. Given the generally intractable objectives, we derive tractable approximations using amortized variational inference parameterized by neural networks.
  arXiv  Detail & Related papers  (2022-11-02T19:37:55Z)
• GCISG: Guided Causal Invariant Learning for Improved Syn-to-real Generalization [1.2215956380648065]
  Training a deep learning model with artificially generated data can be an alternative when training data are scarce. In this paper, we characterize the domain gap by using a causal framework for data generation. We propose causal invariance learning which encourages the model to learn a style-invariant representation that enhances the syn-to-real generalization.
  arXiv  Detail & Related papers  (2022-08-22T02:39:05Z)
• Style Interleaved Learning for Generalizable Person Re-identification [69.03539634477637]
  We propose a novel style interleaved learning (IL) framework for DG ReID training. Unlike conventional learning strategies, IL incorporates two forward propagations and one backward propagation for each iteration. We show that our model consistently outperforms state-of-the-art methods on large-scale benchmarks for DG ReID.
  arXiv  Detail & Related papers  (2022-07-07T07:41:32Z)
• Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
  We propose a novel approach to identify meaningful and independent factors of variation in a dataset. Our method involves two separate latent subspaces for the target property and the remaining input information. We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
  arXiv  Detail & Related papers  (2021-11-25T17:33:12Z)
• CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signals [92.60744099084157]
  We propose differentiable data augmentation amenable to gradient-based learning. We demonstrate the relevance of our approach on the clinically relevant sleep staging classification task.
  arXiv  Detail & Related papers  (2021-06-25T15:28:48Z)
• Hard-label Manifolds: Unexpected Advantages of Query Efficiency for Finding On-manifold Adversarial Examples [67.23103682776049]
  Recent zeroth order hard-label attacks on image classification models have shown comparable performance to their first-order, gradient-level alternatives. It was recently shown in the gradient-level setting that regular adversarial examples leave the data manifold, while their on-manifold counterparts are in fact generalization errors. We propose an information-theoretic argument based on a noisy manifold distance oracle, which leaks manifold information through the adversary's gradient estimate.
  arXiv  Detail & Related papers  (2021-03-04T20:53:06Z)
• Self-Supervised Domain Adaptation with Consistency Training [0.2462953128215087]
  We consider the problem of unsupervised domain adaptation for image classification. We create a self-supervised pretext task by augmenting the unlabeled data with a certain type of transformation. We force the representation of the augmented data to be consistent with that of the original data.
  arXiv  Detail & Related papers  (2020-10-15T06:03:47Z)
• What Should Not Be Contrastive in Contrastive Learning [110.14159883496859]
  We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines on downstream tasks.
  arXiv  Detail & Related papers  (2020-08-13T03:02:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.