Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuning

• URL: http://arxiv.org/abs/2412.16956v2
• Date: Tue, 24 Dec 2024 09:07:26 GMT
• Title: Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuning
• Authors: Haowei Zhu, Fangyuan Zhang, Rui Qin, Tianxiang Pan, Junhai Yong, Bin Wang,
• Abstract summary: Visual Prompt Tuning is noted for its superior performance compared to full fine-tuning. Ship significantly improves performance, achieving a 4.9% gain in accuracy over VPT with a ViT-B/16 backbone on VTAB-1k tasks.
• Score: 13.384550074613717
• License:
• Abstract: As the scale of vision models continues to grow, Visual Prompt Tuning (VPT) has emerged as a parameter-efficient transfer learning technique, noted for its superior performance compared to full fine-tuning. However, indiscriminately applying prompts to every layer without considering their inherent correlations, can cause significant disturbances, leading to suboptimal transferability. Additionally, VPT disrupts the original self-attention structure, affecting the aggregation of visual features, and lacks a mechanism for explicitly mining discriminative visual features, which are crucial for classification. To address these issues, we propose a Semantic Hierarchical Prompt (SHIP) fine-tuning strategy. We adaptively construct semantic hierarchies and use semantic-independent and semantic-shared prompts to learn hierarchical representations. We also integrate attribute prompts and a prompt matching loss to enhance feature discrimination and employ decoupled attention for robustness and reduced inference costs. SHIP significantly improves performance, achieving a 4.9% gain in accuracy over VPT with a ViT-B/16 backbone on VTAB-1k tasks. Our code is available at https://github.com/haoweiz23/SHIP.

Related papers

• Efficient Redundancy Reduction for Open-Vocabulary Semantic Segmentation [36.46163240168576]
  Open-vocabulary semantic segmentation (OVSS) is an open-world task that aims to assign each pixel within an image to a specific class defined by arbitrary text descriptions. Recent advancements in large-scale vision-language models have demonstrated their open-vocabulary understanding capabilities. This study introduces ERR-Seg, a novel framework that effectively reduces redundancy to balance accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-01-29T13:24:53Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts. Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• LOBG:Less Overfitting for Better Generalization in Vision-Language Model [19.890629892640206]
  We propose a framework named LOBG for vision-language models. We use CLIP to filter out fine-grained foreground information that might cause overfitting, thereby guiding prompts with basic visual concepts. Our method significantly improves generalization capability and alleviates overfitting compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-10-14T08:06:21Z)
• Sharing Key Semantics in Transformer Makes Efficient Image Restoration [148.22790334216117]
  Self-attention mechanism, a cornerstone of Vision Transformers (ViTs) tends to encompass all global cues. Small segments of a degraded image, particularly those closely aligned semantically, provide particularly relevant information to aid in the restoration process. We propose boosting IR's performance by sharing the key semantics via Transformer for IR (ie, SemanIR) in this paper.
  arXiv  Detail & Related papers  (2024-05-30T12:45:34Z)
• Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning? [92.23438255540968]
  Visual Prompt Tuning is a parameter-efficient transfer learning technique. We conduct a comprehensive analysis across 19 distinct datasets and tasks. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization.
  arXiv  Detail & Related papers  (2024-01-23T16:48:18Z)
• Enhancing Few-shot CLIP with Semantic-Aware Fine-Tuning [61.902254546858465]
  Methods based on Contrastive Language-Image Pre-training have exhibited promising performance in few-shot adaptation tasks. We propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics.
  arXiv  Detail & Related papers  (2023-11-08T05:18:57Z)
• Semantic Feature Integration network for Fine-grained Visual Classification [5.182627302449368]
  We propose the Semantic Feature Integration network (SFI-Net) to address the above difficulties. By eliminating unnecessary features and reconstructing the semantic relations among discriminative features, our SFI-Net has achieved satisfying performance.
  arXiv  Detail & Related papers  (2023-02-13T07:32:25Z)
• Fine-grained Retrieval Prompt Tuning [149.9071858259279]
  Fine-grained Retrieval Prompt Tuning steers a frozen pre-trained model to perform the fine-grained retrieval task from the perspectives of sample prompt and feature adaptation. Our FRPT with fewer learnable parameters achieves the state-of-the-art performance on three widely-used fine-grained datasets.
  arXiv  Detail & Related papers  (2022-07-29T04:10:04Z)
• Dynamic Feature Regularized Loss for Weakly Supervised Semantic Segmentation [37.43674181562307]
  We propose a new regularized loss which utilizes both shallow and deep features that are dynamically updated. Our approach achieves new state-of-the-art performances, outperforming other approaches by a significant margin with more than 6% mIoU increase.
  arXiv  Detail & Related papers  (2021-08-03T05:11:00Z)
• Feature Fusion Vision Transformer for Fine-Grained Visual Categorization [22.91753200323264]
  We propose a novel pure transformer-based framework Feature Fusion Vision Transformer (FFVT) We aggregate the important tokens from each transformer layer to compensate the local, low-level and middle-level information. We design a novel token selection mod-ule called mutual attention weight selection (MAWS) to guide the network effectively and efficiently towards selecting discriminative tokens.
  arXiv  Detail & Related papers  (2021-07-06T01:48:43Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.