Disentangled Latent Spaces Facilitate Data-Driven Auxiliary Learning

• URL: http://arxiv.org/abs/2310.09278v2
• Date: Fri, 29 Nov 2024 12:47:11 GMT
• Title: Disentangled Latent Spaces Facilitate Data-Driven Auxiliary Learning
• Authors: Geri Skenderi, Luigi Capogrosso, Andrea Toaiari, Matteo Denitto, Franco Fummi, Simone Melzi, Marco Cristani,
• Abstract summary: Auxiliary tasks facilitate learning in situations when data is scarce or the principal task of focus is extremely complex. We propose a novel framework, dubbed Detaux, whereby a weakly supervised disentanglement procedure is used to discover a new unrelated auxiliary classification task. We generate the auxiliary classification task through a clustering procedure on the most disentangled subspace, obtaining a discrete set of labels.
• Score: 14.677411619418319
• License:
• Abstract: Auxiliary tasks facilitate learning in situations when data is scarce or the principal task of focus is extremely complex. This idea is primarily inspired by the improved generalization capability induced by solving multiple tasks simultaneously, which leads to a more robust shared representation. Nevertheless, finding optimal auxiliary tasks is a crucial problem that often requires hand-crafted solutions or expensive meta-learning approaches. In this paper, we propose a novel framework, dubbed Detaux, whereby a weakly supervised disentanglement procedure is used to discover a new unrelated auxiliary classification task, which allows us to go from a Single-Task Learning (STL) to a Multi-Task Learning (MTL) problem. The disentanglement procedure works at the representation level, isolating the variation related to the principal task into an isolated subspace and additionally producing an arbitrary number of orthogonal subspaces, each one of them encouraging high separability among the projections. We generate the auxiliary classification task through a clustering procedure on the most disentangled subspace, obtaining a discrete set of labels. Subsequently, the original data, the labels associated with the principal task, and the newly discovered ones can be fed into any MTL framework. Experimental validation on both synthetic and real data, along with various ablation studies, demonstrate promising results, revealing the potential in what has been, so far, an unexplored connection between learning disentangled representations and MTL. The source code will be made available upon acceptance.

Related papers

• Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution [23.375053899418504]
  Instead of probing frozen representations from a complex source task, we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks proving most beneficial.
  arXiv  Detail & Related papers  (2025-01-31T17:12:53Z)
• Mitigating Interference in the Knowledge Continuum through Attention-Guided Incremental Learning [17.236861687708096]
  Attention-Guided Incremental Learning' (AGILE) is a rehearsal-based CL approach that incorporates compact task attention to effectively reduce interference between tasks. AGILE significantly improves generalization performance by mitigating task interference and outperforming rehearsal-based approaches in several CL scenarios.
  arXiv  Detail & Related papers  (2024-05-22T20:29:15Z)
• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Leveraging sparse and shared feature activations for disentangled representation learning [112.22699167017471]
  We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. We validate our approach on six real world distribution shift benchmarks, and different data modalities.
  arXiv  Detail & Related papers  (2023-04-17T01:33:24Z)
• Learning Multi-Tasks with Inconsistent Labels by using Auxiliary Big Task [24.618094251341958]
  Multi-task learning is to improve the performance of the model by transferring and exploiting common knowledge among tasks. We propose a framework to learn these tasks by jointly leveraging both abundant information from a learnt auxiliary big task with sufficiently many classes to cover those of all these tasks. Our experimental results demonstrate its effectiveness in comparison with the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-07T02:46:47Z)
• Transfer Learning in Conversational Analysis through Reusing Preprocessing Data as Supervisors [52.37504333689262]
  Using noisy labels in single-task learning increases the risk of over-fitting. Auxiliary tasks could improve the performance of the primary task learning during the same training.
  arXiv  Detail & Related papers  (2021-12-02T08:40:42Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Pareto Self-Supervised Training for Few-Shot Learning [5.6715399725748155]
  Few-shot auxiliary learning has recently drawn much attention to deal with few labeled data. It is challenging to select a proper weight to balance tasks and reduce task conflict.
  arXiv  Detail & Related papers  (2021-04-16T01:26:25Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.