CVPT: Cross Visual Prompt Tuning

• URL: http://arxiv.org/abs/2408.14961v2
• Date: Sat, 19 Jul 2025 15:15:13 GMT
• Title: CVPT: Cross Visual Prompt Tuning
• Authors: Lingyun Huang, Jianxu Mao, Junfei Yi, Ziming Tao, Yaonan Wang,
• Abstract summary: Cross Visual Prompt Tuning (CVPT) is a cross-attention module to model interactions between prompts and image tokens.<n>CVPT achieves over 4% higher average accuracy, rivaling leading adapter-based methods in both performance and efficiency.<n>Our work confirms that prompt-based methods can achieve exceptional results in visual fine-tuning.
• Score: 15.642102189777072
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) has emerged to mitigate the computational demands of large-scale models. Within computer vision, adapter-based PEFT methods are often favored over prompt-based approaches like Visual Prompt Tuning (VPT) due to the latter's performance and efficiency limitations. Our analysis reveals that VPT's shortcomings stem from its prompt deployment strategy, which can distort the model's inherent self-attention mechanism. To address this, we propose Cross Visual Prompt Tuning (CVPT). CVPT introduces a cross-attention module to directly model interactions between prompts and image tokens. This design decouples the prompts from the input sequence, preserving the original self-attention integrity while enabling efficient feature integration. Furthermore, we employ a weight-sharing mechanism for cross-attention initialization, which enhances representative capability without a large parameter overhead. Extensive experiments across 25 datasets show that CVPT significantly outperforms VPT. For instance, on the VTAB-1K benchmark, CVPT achieves over 4% higher average accuracy, rivaling leading adapter-based methods in both performance and efficiency. Our work confirms that prompt-based methods can achieve exceptional results in visual fine-tuning. The code is available at https://github.com/Lingyun0419/CVPT

Related papers

• Visual Instance-aware Prompt Tuning [21.538712755298413]
  Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers.<n>We propose Visual Instance-aware Prompt Tuning (ViaPT), which generates instance-aware prompts based on each individual input.<n>ViaPT overcomes limitations by balancing dataset-level and instance-level knowledge, while reducing the amount of learnable parameters.
  arXiv  Detail & Related papers  (2025-07-10T14:23:15Z)
• Adaptive Prompt: Unlocking the Power of Visual Prompt Tuning [27.703316805290843]
  Visual Prompt Tuning (VPT) has emerged as a powerful method for adapting pre-trained vision models to downstream tasks. We propose Visual Adaptive Prompt Tuning (VAPT), a new generation of prompts that redefines prompts as adaptive functions of the input. Our theoretical analysis shows that VAPT achieves optimal sample efficiency.
  arXiv  Detail & Related papers  (2025-01-31T07:41:06Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation [67.13876021157887]
  Dynamic Tuning (DyT) is a novel approach to improve both parameter and inference efficiency for ViT adaptation. DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.
  arXiv  Detail & Related papers  (2024-03-18T14:05:52Z)
• Revisiting the Power of Prompt for Visual Tuning [50.11465784194896]
  This study explores the correlation evolvement between prompts and patch tokens during proficient training. Inspired by the observation that the prompt tokens tend to share high mutual information with patch tokens, we propose initializing prompts with downstream token prototypes. Our method significantly advances the adaptation for self-supervised pretraining, achieving impressive task performance gains of at least 10% to 30%.
  arXiv  Detail & Related papers  (2024-02-04T07:49:02Z)
• Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning? [92.23438255540968]
  Visual Prompt Tuning is a parameter-efficient transfer learning technique. We conduct a comprehensive analysis across 19 distinct datasets and tasks. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization.
  arXiv  Detail & Related papers  (2024-01-23T16:48:18Z)
• E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning [55.50908600818483]
  Fine-tuning large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. We propose an Effective and Efficient Visual Prompt Tuning (E2VPT) approach for large-scale transformer-based model adaptation. Our approach outperforms several state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2023-07-25T19:03:21Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Visual Prompt Tuning for Test-time Domain Adaptation [48.16620171809511]
  We propose a simple recipe called data-efficient prompt tuning (DePT) with two key ingredients. We find such parameter-efficient finetuning can efficiently adapt the model representation to the target domain without overfitting to the noise in the learning objective. With much fewer parameters, DePT demonstrates not only state-of-the-art performance on major adaptation benchmarks, but also superior data efficiency.
  arXiv  Detail & Related papers  (2022-10-10T16:45:13Z)
• Visual Prompt Tuning [74.5309408185523]
  This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen.
  arXiv  Detail & Related papers  (2022-03-23T01:17:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.