Visual prompting reimagined: The power of the Activation Prompts

• URL: http://arxiv.org/abs/2604.06440v1
• Date: Tue, 07 Apr 2026 20:28:24 GMT
• Title: Visual prompting reimagined: The power of the Activation Prompts
• Abstract summary: We introduce a concept, termed activation prompt (AP), which extends the scope of the input-level VP.<n>AP is closely related to normalization tuning in convolutional neural networks and vision transformers.<n>We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers.
• Score: 72.85146015928626
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual prompting (VP) has emerged as a popular method to repurpose pretrained vision models for adaptation to downstream tasks. Unlike conventional model fine-tuning techniques, VP introduces a universal perturbation directly into the input data to facilitate task-specific fine-tuning rather than modifying model parameters. However, there exists a noticeable performance gap between VP and conventional fine-tuning methods, highlighting an unexplored realm in theory and practice to understand and advance the input-level VP to reduce its current performance gap. Towards this end, we introduce a generalized concept, termed activation prompt (AP), which extends the scope of the input-level VP by enabling universal perturbations to be applied to activation maps within the intermediate layers of the model. By using AP to revisit the problem of VP and employing it as an analytical tool, we demonstrate the intrinsic limitations of VP in both performance and efficiency, revealing why input-level prompting may lack effectiveness compared to AP, which exhibits a model-dependent layer preference. We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers, although each model type has distinct layer preferences for prompting. We also theoretically elucidate the rationale behind such a preference by analyzing global features across layers. Through extensive experiments across 29 datasets and various model architectures, we provide a comprehensive performance analysis of AP, comparing it with VP and parameter-efficient fine-tuning baselines. Our results demonstrate AP's superiority in both accuracy and efficiency, considering factors such as time, parameters, memory usage, and throughput.

Related papers

• APEX: Probing Neural Networks via Activation Perturbation [10.517751599566548]
  We introduce Activation Perturbation for EXploration (APEX) as an inference-time probing paradigm for neural networks.<n>APEX perturbs hidden activations while keeping both inputs and model parameters fixed.<n>Our results show that APEX offers an effective perspective for exploring, and understanding neural networks beyond what is accessible from input space alone.
  arXiv  Detail & Related papers  (2026-02-03T14:36:36Z)
• Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation [0.9137554315375919]
  Visual prompting (VP) has emerged as a promising parameter-efficient fine-tuning approach for adapting pre-trained vision models to downstream tasks.<n>We propose ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP's expressive power by introducing complementary transformation operations.<n>We show that ACAVP achieves state-of-the-art accuracy among VP methods, surpasses linear probing in average accuracy, and exhibits superior robustness to distribution shifts.
  arXiv  Detail & Related papers  (2025-10-09T06:08:15Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization [69.33162366130887]
  Domain Generalization (DG) endeavors to create machine learning models that excel in unseen scenarios by learning invariant features. We introduce a novel method designed to supplement the model with domain-level and task-specific characteristics. This approach aims to guide the model in more effectively separating invariant features from specific characteristics, thereby boosting the generalization.
  arXiv  Detail & Related papers  (2024-01-18T04:23:21Z)
• AutoVP: An Automated Visual Prompting Framework and Benchmark [66.5618543577204]
  Visual prompting (VP) is an emerging parameter-efficient fine-tuning approach to adapting pre-trained vision models to solve various downstream image-classification tasks. We propose AutoVP, an end-to-end expandable framework for automating VP design choices, along with 12 downstream image-classification tasks. Our experimental results show that AutoVP outperforms the best-known current VP methods by a substantial margin.
  arXiv  Detail & Related papers  (2023-10-12T14:55:31Z)
• Uncovering the Hidden Cost of Model Compression [43.62624133952414]
  Visual Prompting has emerged as a pivotal method for transfer learning in computer vision. Model compression detrimentally impacts the performance of visual prompting-based transfer. However, negative effects on calibration are not present when models are compressed via quantization.
  arXiv  Detail & Related papers  (2023-08-29T01:47:49Z)
• FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning [9.950367271170592]
  We investigate where and how to partially personalize a Vision Transformers (ViT) model. Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix. We evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness compared to several advanced PFL methods.
  arXiv  Detail & Related papers  (2023-08-17T19:22:30Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Trading Information between Latents in Hierarchical Variational Autoencoders [8.122270502556374]
  Variational Autoencoders (VAEs) were originally motivated as probabilistic generative models in which one performs approximate Bayesian inference. The proposal of $beta$-VAEs breaks this interpretation and generalizes VAEs to application domains beyond generative modeling. We identify a general class of inference models for which one can split the rate into contributions from each layer, which can then be tuned independently.
  arXiv  Detail & Related papers  (2023-02-09T18:56:11Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.