ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

• URL: http://arxiv.org/abs/2608.21194v1
• Date: Fri, 21 Aug 2026 15:03:51 GMT
• Title: ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation
• Abstract summary: Energy-Shaped Visual Prompting (ES-VP) is a novel approach that generates image-specific prompts.<n>ES-VP directly utilizes the pre-trained model for adaptive prompt generation, ensuring both parameter efficiency and improved generalization.
• Score: 39.09241645535107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual prompting (VP) has emerged as a parameter-efficient method for adapting pre-trained models to downstream tasks. However, existing approaches encounter a trade-off between flexibility and efficiency. Some methods apply a fixed prompt to all images, ignoring individual image characteristics, while others introduce auxiliary networks to generate diverse prompts. Although the latter can improve performance, it also significantly increases parameter usage and the potential for overfitting to specific datasets. Furthermore, the auxiliary networks, combined with inherent biases in pre-trained models, limit scalability and generalization. In this paper, we propose Energy-Shaped Visual Prompting (ES-VP), a novel approach that generates image-specific prompts using low-rank initialization and energy-guided dynamic adaptation, achieving superior performance with fewer parameters compared to single-prompt methods. ES-VP directly utilizes the pre-trained model for adaptive prompt generation, ensuring both parameter efficiency and improved generalization. Extensive experiments conducted on five architectures across fifteen datasets demonstrate that ES-VP consistently outperforms current state-of-the-art (SOTA) single and diverse VP methods. For instance, using the CLIP architecture across four datasets, ES-VP outperforms the SOTA method DAM-VP by an average of 2.6\% in accuracy while utilizing 590$\times$ fewer VP parameters, thereby establishing a new benchmark for efficient and generalizable model adaptation.

Related papers

• Visual prompting reimagined: The power of the Activation Prompts [72.85146015928626]
  We introduce a concept, termed activation prompt (AP), which extends the scope of the input-level VP.<n>AP is closely related to normalization tuning in convolutional neural networks and vision transformers.<n>We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers.
  arXiv  Detail & Related papers  (2026-04-07T20:28:24Z)
• Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation [0.9137554315375919]
  Visual prompting (VP) has emerged as a promising parameter-efficient fine-tuning approach for adapting pre-trained vision models to downstream tasks.<n>We propose ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP's expressive power by introducing complementary transformation operations.<n>We show that ACAVP achieves state-of-the-art accuracy among VP methods, surpasses linear probing in average accuracy, and exhibits superior robustness to distribution shifts.
  arXiv  Detail & Related papers  (2025-10-09T06:08:15Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• CVPT: Cross Visual Prompt Tuning [15.642102189777072]
  Cross Visual Prompt Tuning (CVPT) is a cross-attention module to model interactions between prompts and image tokens.<n>CVPT achieves over 4% higher average accuracy, rivaling leading adapter-based methods in both performance and efficiency.<n>Our work confirms that prompt-based methods can achieve exceptional results in visual fine-tuning.
  arXiv  Detail & Related papers  (2024-08-27T11:07:19Z)
• Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition [22.615830919860777]
  This paper presents an efficient visual recognition paradigm, called Dynamic Adapter (Dyn-Adapter) We devise a dynamic architecture with balanced early heads for multi-level feature extraction, along with adaptive training strategy. We reduce FLOPs during inference by 50%, while maintaining or even yielding higher recognition accuracy.
  arXiv  Detail & Related papers  (2024-07-19T13:33:38Z)
• Parameter-Efficient Fine-Tuning With Adapters [5.948206235442328]
  This research introduces a novel adaptation method utilizing the UniPELT framework as a base. Our method employs adapters, which enable efficient transfer of pretrained models to new tasks with minimal retraining of the base model parameters.
  arXiv  Detail & Related papers  (2024-05-09T01:40:38Z)
• Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation [67.13876021157887]
  Dynamic Tuning (DyT) is a novel approach to improve both parameter and inference efficiency for ViT adaptation. DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.
  arXiv  Detail & Related papers  (2024-03-18T14:05:52Z)
• E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning [55.50908600818483]
  Fine-tuning large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. We propose an Effective and Efficient Visual Prompt Tuning (E2VPT) approach for large-scale transformer-based model adaptation. Our approach outperforms several state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2023-07-25T19:03:21Z)
• Visual Prompt Tuning [74.5309408185523]
  This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen.
  arXiv  Detail & Related papers  (2022-03-23T01:17:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.