CrossPT: Exploring Cross-Task Transferability through Multi-Task Prompt Tuning

• URL: http://arxiv.org/abs/2509.14253v1
• Date: Thu, 11 Sep 2025 05:13:28 GMT
• Title: CrossPT: Exploring Cross-Task Transferability through Multi-Task Prompt Tuning
• Abstract summary: Cross-task Prompt Tuning (CrossPT) is a modular framework for multi-task prompt tuning.<n>CrossPT decomposes each target prompt into shared, pre-trained source prompts and task-specific private prompts, combined via a learned attention mechanism.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt tuning offers a parameter-efficient way to adapt large pre-trained language models to new tasks, but most existing approaches are designed for single-task settings, failing to share knowledge across related tasks. We propose Cross-task Prompt Tuning (CrossPT), a modular framework for multi-task prompt tuning that enables controlled knowledge transfer while maintaining task-specific specialization. CrossPT decomposes each target prompt into shared, pre-trained source prompts and task-specific private prompts, combined via a learned attention mechanism. To support robust transfer, we systematically investigate key design factors including prompt initialization, balancing shared and private prompts, number of source prompts, learning rates, task prefixes, and label semantics. Empirical results on GLUE and related benchmarks show that CrossPT achieves higher accuracy and robustness compared to traditional prompt tuning and related methods, particularly in low-resource scenarios, while maintaining strong parameter efficiency.

Related papers

• T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler [51.05812650668468]
  Multi-task reinforcement learning (MTRL) is a technique to train multiple tasks simultaneously, where previous works usually train a single model to solve different tasks by sharing parameters across various tasks.<n>We propose a novel MTRL framework called Task-Specific feature Selector and Scheduler (T3S), which consists of two components: a feature selector and a task scheduler.<n>Specifically, the feature selectors employ hypernetworks to construct task-specific soft masks, which can be applied by globally shared representation to construct task-specific features.
  arXiv  Detail & Related papers  (2026-08-31T13:30:06Z)
• Task-Anchored Representation Shaping for Pre-Trained Model-Based Continual Learning [15.779738807521618]
  We propose Task-Anchored Inference Latent Shaping (TAILS) as a lightweight post-PTM module for continuous learning.<n>TAILS uses fixed task anchors as persistent references to accumulated knowledge, then composes relevant evidence across tasks into latent recall.<n>It can improve classification and task-inference performance with modest parameter overhead and negligible inference cost.
  arXiv  Detail & Related papers  (2026-08-17T09:51:21Z)
• One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning [52.966712416640085]
  We propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies.<n>SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-29T08:54:58Z)
• Enhancing Few-Shot Transfer Learning with Optimized Multi-Task Prompt Tuning through Modular Prompt Composition [0.0]
  Multi-task prompt tuning has garnered considerable attention for its inherent modularity and potential to enhance parameter-efficient transfer learning.<n>This paper aims to analyze and improve the performance of multiple tasks by facilitating the transfer of knowledge between their corresponding prompts in a multi-task setting.
  arXiv  Detail & Related papers  (2024-08-23T17:01:51Z)
• PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer [76.39111896665585]
  Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
  arXiv  Detail & Related papers  (2024-07-04T10:37:58Z)
• Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning [44.43258626098661]
  We argue that when we extract knowledge from source tasks via training source prompts, we need to consider this correlation among source tasks for better transfer to target tasks. We propose a Bayesian approach where we work with the posterior distribution of prompts across source tasks. We show extensive experimental results on the standard benchmark NLP tasks, where our Bayesian multi-task transfer learning approach outperforms the state-of-the-art methods in many settings.
  arXiv  Detail & Related papers  (2024-02-13T16:57:02Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning [43.639430661322585]
  We propose multitask prompt tuning (MPT) MPT learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task.
  arXiv  Detail & Related papers  (2023-03-06T03:25:59Z)
• Multitask Vision-Language Prompt Tuning [103.5967011236282]
  We propose multitask vision-language prompt tuning (MV) MV incorporates cross-task knowledge into prompt tuning for vision-language models. Results in 20 vision tasks demonstrate that the proposed approach outperforms all single-task baseline prompt tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T18:41:44Z)
• Prompt Tuning with Soft Context Sharing for Vision-Language Models [42.61889428498378]
  We propose a novel method to tune pre-trained vision-language models on multiple target few-shot tasks jointly. We show that SoftCPT significantly outperforms single-task prompt tuning methods.
  arXiv  Detail & Related papers  (2022-08-29T10:19:10Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• Attentional Mixtures of Soft Prompt Tuning for Parameter-efficient Multi-task Knowledge Sharing [53.399742232323895]
  ATTEMPT is a new modular, multi-task, and parameter-efficient language model (LM) tuning approach. It combines knowledge transferred across different tasks via a mixture of soft prompts while keeping original LM unchanged. It is parameter-efficient (e.g., updates 1,600 times fewer parameters than fine-tuning) and enables multi-task learning and flexible extensions.
  arXiv  Detail & Related papers  (2022-05-24T10:48:33Z)
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer [7.2462572989580405]
  We propose a novel prompt-based transfer learning approach called SPoT: Soft Prompt Transfer. We show SPoT significantly boosts the performance of PromptTuning across many tasks. We also conduct a large-scale study on task transferability with 26 NLP tasks and 160 combinations of source-target tasks.
  arXiv  Detail & Related papers  (2021-10-15T07:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.