MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representations

• URL: http://arxiv.org/abs/2305.17191v2
• Date: Fri, 26 Jan 2024 14:44:42 GMT
• Title: MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representations
• Authors: Calum Heggan, Tim Hospedales, Sam Budgett, Mehrdad Yaghoobi
• Abstract summary: We propose a multi-task self-supervised framework (MT-SLVR) that learns both variant and invariant features in a parameter-efficient manner. We evaluate our approach on few-shot classification tasks drawn from a variety of audio domains and demonstrate improved classification performance.
• Score: 2.94944680995069
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Contrastive self-supervised learning has gained attention for its ability to create high-quality representations from large unlabelled data sets. A key reason that these powerful features enable data-efficient learning of downstream tasks is that they provide augmentation invariance, which is often a useful inductive bias. However, the amount and type of invariances preferred is not known apriori, and varies across different downstream tasks. We therefore propose a multi-task self-supervised framework (MT-SLVR) that learns both variant and invariant features in a parameter-efficient manner. Our multi-task representation provides a strong and flexible feature that benefits diverse downstream tasks. We evaluate our approach on few-shot classification tasks drawn from a variety of audio domains and demonstrate improved classification performance on all of them

Related papers

• Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning [24.36351102003414]
  Human activity recognition (HAR) is a time series classification task that focuses on identifying the motion patterns from human sensor readings. We propose a novel approach called Diverse and Discriminative representation Learning (DDLearn) for generalizable lowresource HAR. Our method significantly outperforms state-of-art methods by an average accuracy improvement of 9.5%.
  arXiv  Detail & Related papers  (2023-05-25T08:24:22Z)
• Leveraging sparse and shared feature activations for disentangled representation learning [112.22699167017471]
  We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. We validate our approach on six real world distribution shift benchmarks, and different data modalities.
  arXiv  Detail & Related papers  (2023-04-17T01:33:24Z)
• Amortised Invariance Learning for Contrastive Self-Supervision [11.042648980854485]
  We introduce the notion of amortised invariance learning for contrastive self supervision. We show that our amortised features provide a reliable way to learn diverse downstream tasks with different invariance requirements. This provides an exciting perspective that opens up new horizons in the field of general purpose representation learning.
  arXiv  Detail & Related papers  (2023-02-24T16:15:11Z)
• Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
  Multi-task learning aims to acquire a set of functions that perform well for diverse tasks. In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks. We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
  arXiv  Detail & Related papers  (2022-10-27T16:06:47Z)
• Improving VAE-based Representation Learning [26.47244578124654]
  We study what properties are required for good representations and how different VAE structure choices could affect the learned properties. We show that by using a decoder that prefers to learn local features, the remaining global features can be well captured by the latent.
  arXiv  Detail & Related papers  (2022-05-28T23:00:18Z)
• Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports [2.5953185061765884]
  We show that a single multi-tasking model can match the performance of task specific models. We validate our observations on our internal radiologist-annotated datasets on the cervical and lumbar spine.
  arXiv  Detail & Related papers  (2022-05-06T01:51:19Z)
• Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization [76.68866368409216]
  We propose learning to dynamically select discretization tightness conditioned on inputs. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
  arXiv  Detail & Related papers  (2022-02-02T23:54:26Z)
• Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks [79.13089902898848]
  Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. We show that different tasks in computer vision require features to encode different (in)variances.
  arXiv  Detail & Related papers  (2021-11-22T18:16:35Z)
• Learning Invariant Representations across Domains and Tasks [81.30046935430791]
  We propose a novel Task Adaptation Network (TAN) to solve this unsupervised task transfer problem. In addition to learning transferable features via domain-adversarial training, we propose a novel task semantic adaptor that uses the learning-to-learn strategy to adapt the task semantics. TAN significantly increases the recall and F1 score by 5.0% and 7.8% compared to recently strong baselines.
  arXiv  Detail & Related papers  (2021-03-03T11:18:43Z)
• What Should Not Be Contrastive in Contrastive Learning [110.14159883496859]
  We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines on downstream tasks.
  arXiv  Detail & Related papers  (2020-08-13T03:02:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.