Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning

• URL: http://arxiv.org/abs/2407.16245v1
• Date: Tue, 23 Jul 2024 07:31:43 GMT
• Title: Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning
• Authors: Pin-Jie Lin, Miaoran Zhang, Marius Mosbach, Dietrich Klakow,
• Abstract summary: We show that the transfer performance exhibits severe variance across different source tasks and training seeds. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability. We introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction.
• Score: 21.652389166495407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. In this work, we experiment with 130 source-target task combinations and demonstrate that the transfer performance exhibits severe variance across different source tasks and training seeds, highlighting the crucial role of intermediate-task selection in a broader context. We compare four representative task selection methods in a unified setup, focusing on their effectiveness and consistency. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability by improving task prediction scores from 2.59% to 3.96%. Despite their strong performance, we observe that the task embeddings do not consistently demonstrate superiority for tasks requiring reasoning abilities. Furthermore, we introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction. Our findings suggest that token-wise similarity is better predictive for predicting transferability compared to averaging weights.

Related papers

• Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks [51.15473776489712]
  We introduce a simple yet effective task selection method that leverages instruction information alone to identify relevant tasks. Our method is significantly more efficient than traditional approaches, which require complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. Experimental results demonstrate that training on a small set of tasks, chosen solely on the instructions, results in substantial improvements in performance on benchmarks such as P3, Big-Bench, NIV2, and Big-Bench Hard.
  arXiv  Detail & Related papers  (2024-04-25T08:49:47Z)
• Understanding the Transferability of Representations via Task-Relatedness [8.425690424016986]
  We propose a novel analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. Our experiments using state-of-the-art pre-trained models show the effectiveness of task-relatedness in explaining transferability on various vision and language tasks.
  arXiv  Detail & Related papers  (2023-07-03T08:06:22Z)
• Divergence-Based Domain Transferability for Zero-Shot Classification [78.55044112903148]
  Transferring learned patterns from pretrained neural language models has been shown to significantly improve effectiveness across a variety of language-based tasks. Further tuning on intermediate tasks has been demonstrated to provide additional performance benefits, provided the intermediate task is sufficiently related to the target task. However, how to identify related tasks is an open problem, and brute-force searching effective task combinations is prohibitively expensive.
  arXiv  Detail & Related papers  (2023-02-11T16:04:38Z)
• Identifying Suitable Tasks for Inductive Transfer Through the Analysis of Feature Attributions [78.55044112903148]
  We use explainability techniques to predict whether task pairs will be complementary, through comparison of neural network activation between single-task models. Our results show that, through this approach, it is possible to reduce training time by up to 83.5% at a cost of only 0.034 reduction in positive-class F1 on the TREC-IS 2020-A dataset.
  arXiv  Detail & Related papers  (2022-02-02T15:51:07Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Measuring and Harnessing Transference in Multi-Task Learning [58.48659733262734]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We analyze the dynamics of information transfer, or transference, across tasks throughout training.
  arXiv  Detail & Related papers  (2020-10-29T08:25:43Z)
• Exploring and Predicting Transferability across NLP Tasks [115.6278033699853]
  We study the transferability between 33 NLP tasks across three broad classes of problems. Our results show that transfer learning is more beneficial than previously thought. We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task.
  arXiv  Detail & Related papers  (2020-05-02T09:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.