Pareto Self-Supervised Training for Few-Shot Learning
- URL: http://arxiv.org/abs/2104.07841v2
- Date: Mon, 19 Apr 2021 02:55:31 GMT
- Title: Pareto Self-Supervised Training for Few-Shot Learning
- Authors: Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, Donglin Wang
- Abstract summary: Few-shot auxiliary learning has recently drawn much attention to deal with few labeled data.
It is challenging to select a proper weight to balance tasks and reduce task conflict.
- Score: 5.6715399725748155
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While few-shot learning (FSL) aims for rapid generalization to new concepts
with little supervision, self-supervised learning (SSL) constructs supervisory
signals directly computed from unlabeled data. Exploiting the complementarity
of these two manners, few-shot auxiliary learning has recently drawn much
attention to deal with few labeled data. Previous works benefit from sharing
inductive bias between the main task (FSL) and auxiliary tasks (SSL), where the
shared parameters of tasks are optimized by minimizing a linear combination of
task losses. However, it is challenging to select a proper weight to balance
tasks and reduce task conflict. To handle the problem as a whole, we propose a
novel approach named as Pareto self-supervised training (PSST) for FSL. PSST
explicitly decomposes the few-shot auxiliary problem into multiple constrained
multi-objective subproblems with different trade-off preferences, and here a
preference region in which the main task achieves the best performance is
identified. Then, an effective preferred Pareto exploration is proposed to find
a set of optimal solutions in such a preference region. Extensive experiments
on several public benchmark datasets validate the effectiveness of our approach
by achieving state-of-the-art performance.
Related papers
- Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation [69.60321475454843]
We propose DPCPL, the first pre-training and prompt-tuning paradigm tailored for Multi-Behavior Sequential Recommendation.
In the pre-training stage, we propose a novel Efficient Behavior Miner (EBM) to filter out the noise at multiple time scales.
Subsequently, we propose to tune the pre-trained model in a highly efficient manner with the proposed Customized Prompt Learning (CPL) module.
arXiv Detail & Related papers (2024-08-21T06:48:38Z) - Efficient Pareto Manifold Learning with Low-Rank Structure [31.082432589391953]
Multi-task learning is inherently a multi-objective optimization problem.
We propose a novel approach that integrates a main network with several low-rank matrices.
It significantly reduces the number of parameters and facilitates the extraction of shared features.
arXiv Detail & Related papers (2024-07-30T11:09:27Z) - Mitigating Interference in the Knowledge Continuum through Attention-Guided Incremental Learning [17.236861687708096]
Attention-Guided Incremental Learning' (AGILE) is a rehearsal-based CL approach that incorporates compact task attention to effectively reduce interference between tasks.
AGILE significantly improves generalization performance by mitigating task interference and outperforming rehearsal-based approaches in several CL scenarios.
arXiv Detail & Related papers (2024-05-22T20:29:15Z) - Disentangled Latent Spaces Facilitate Data-Driven Auxiliary Learning [14.677411619418319]
Auxiliary tasks facilitate learning in situations when data is scarce or the principal task of focus is extremely complex.
We propose a novel framework, dubbed Detaux, whereby a weakly supervised disentanglement procedure is used to discover a new unrelated auxiliary classification task.
We generate the auxiliary classification task through a clustering procedure on the most disentangled subspace, obtaining a discrete set of labels.
arXiv Detail & Related papers (2023-10-13T17:40:39Z) - Task Selection and Assignment for Multi-modal Multi-task Dialogue Act
Classification with Non-stationary Multi-armed Bandits [11.682678945754837]
Multi-task learning (MTL) aims to improve the performance of a primary task by jointly learning with related auxiliary tasks.
Previous studies suggest that such a random selection of tasks may not be helpful, and can even be harmful to performance.
This paper proposes a method for selecting and assigning tasks based on non-stationary multi-armed bandits.
arXiv Detail & Related papers (2023-09-18T14:51:51Z) - Auxiliary Learning as an Asymmetric Bargaining Game [50.826710465264505]
We propose a novel approach, named AuxiNash, for balancing tasks in auxiliary learning.
We describe an efficient procedure for learning the bargaining power of tasks based on their contribution to the performance of the main task.
We evaluate AuxiNash on multiple multi-task benchmarks and find that it consistently outperforms competing methods.
arXiv Detail & Related papers (2023-01-31T09:41:39Z) - Pareto Manifold Learning: Tackling multiple tasks via ensembles of
single-task models [50.33956216274694]
In Multi-Task Learning (MTL), tasks may compete and limit the performance achieved on each other, rather than guiding the optimization to a solution.
We propose textitPareto Manifold Learning, an ensembling method in weight space.
arXiv Detail & Related papers (2022-10-18T11:20:54Z) - Decoupled Adversarial Contrastive Learning for Self-supervised
Adversarial Robustness [69.39073806630583]
Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields.
We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
arXiv Detail & Related papers (2022-07-22T06:30:44Z) - Multi-Task Learning as a Bargaining Game [63.49888996291245]
In Multi-task learning (MTL), a joint model is trained to simultaneously make predictions for several tasks.
Since the gradients of these different tasks may conflict, training a joint model for MTL often yields lower performance than its corresponding single-task counterparts.
We propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
arXiv Detail & Related papers (2022-02-02T13:21:53Z) - Task-Feature Collaborative Learning with Application to Personalized
Attribute Prediction [166.87111665908333]
We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL)
Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks.
As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
arXiv Detail & Related papers (2020-04-29T02:32:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.