All You Need is One: Capsule Prompt Tuning with a Single Vector

• URL: http://arxiv.org/abs/2510.16670v1
• Date: Sun, 19 Oct 2025 00:02:59 GMT
• Title: All You Need is One: Capsule Prompt Tuning with a Single Vector
• Authors: Yiyang Liu, James C. Liang, Heng Fan, Wenhao Yang, Yiming Cui, Xiaotian Han, Lifu Huang, Dongfang Liu, Qifan Wang, Cheng Han,
• Abstract summary: Current prompt-based learning methods rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts.<n>We introduce Capsule Prompt-Tuning (CaPT), an efficient and effective solution that leverages off-the-shelf, informative instance semantics into prompt-based learning.<n>Our approach innovatively integrates both instance-aware and task-aware information in a nearly parameter-free manner.
• Score: 86.68105855537762
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt-based learning has emerged as a parameter-efficient finetuning (PEFT) approach to facilitate Large Language Model (LLM) adaptation to downstream tasks by conditioning generation with task-aware guidance. Despite its successes, current prompt-based learning methods heavily rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts, introducing additional computational burden. Worse yet, our pioneer findings indicate that the task-aware prompt design is inherently limited by its absence of instance-aware information, leading to a subtle attention interplay with the input sequence. In contrast, simply incorporating instance-aware information as a part of the guidance can enhance the prompt-tuned model performance without additional fine-tuning. Moreover, we find an interesting phenomenon, namely "attention anchor", that incorporating instance-aware tokens at the earliest position of the sequence can successfully preserve strong attention to critical structural information and exhibit more active attention interaction with all input tokens. In light of our observation, we introduce Capsule Prompt-Tuning (CaPT), an efficient and effective solution that leverages off-the-shelf, informative instance semantics into prompt-based learning. Our approach innovatively integrates both instance-aware and task-aware information in a nearly parameter-free manner (i.e., one single capsule prompt). Empirical results demonstrate that our method can exhibit superior performance across various language tasks (e.g., 84.03\% average accuracy on T5-Large), serving as an "attention anchor," while enjoying high parameter efficiency (e.g., 0.003\% of model parameters on Llama3.2-1B).

Related papers

• Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learning [76.32953653161417]
  Class-incremental learning enables models to learn new classes progressively while preserving knowledge of previously learned ones.<n>Recent advances in this field have shifted towards parameter-efficient fine-tuning techniques.<n>We present a novel prompt-based approach that addresses the limitation of current approaches.
  arXiv  Detail & Related papers  (2025-03-11T02:27:37Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding [51.48361798508375]
  We develop an information-theoretic framework that formulates soft prompt tuning as maximizing mutual information between prompts and other model parameters. We show that InfoPrompt can significantly accelerate the convergence of the prompt tuning and outperform traditional prompt tuning methods.
  arXiv  Detail & Related papers  (2023-06-08T04:31:48Z)
• On the Role of Attention in Prompt-tuning [90.97555030446563]
  We study prompt-tuning for one-layer attention architectures and study contextual mixture-models. We show that softmax-prompt-attention is provably more expressive than softmax-self-attention and linear-prompt-attention. We also provide experiments that verify our theoretical insights on real datasets and demonstrate how prompt-tuning enables the model to attend to context-relevant information.
  arXiv  Detail & Related papers  (2023-06-06T06:23:38Z)
• Dynamic Prompting: A Unified Framework for Prompt Tuning [33.175097465669374]
  We present a unified dynamic prompt (DP) tuning strategy that dynamically determines different factors of prompts based on specific tasks and instances. Experimental results underscore the significant performance improvement achieved by dynamic prompt tuning across a wide range of tasks. We establish the universal applicability of our approach under full-data, few-shot, and multitask scenarios.
  arXiv  Detail & Related papers  (2023-03-06T06:04:46Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.