Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models

• URL: http://arxiv.org/abs/2508.02671v1
• Date: Mon, 04 Aug 2025 17:59:56 GMT
• Title: Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models
• Abstract summary: Augmentation-driven Prompt Tuning (AugPT) is a self-contained distillation-based prompt tuning approach.<n>AugPT employs self-supervised augmentation on unlabeled images in the training set, and introduces a novel gating mechanism based on consensus test.
• Score: 40.43692693102914
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: For CLIP-based prompt tuning, introducing more data as additional knowledge for enhancing fine-tuning process is proved to be an effective approach. Existing data amplification strategies for prompt tuning typically rely on external knowledge (e.g., large language models or pre-structured knowledge bases), resulting in higher costs for data collection and processing, while generally ignoring further utilization of features in image modality. To address this, we propose Augmentation-driven Prompt Tuning (AugPT), a self-contained distillation-based prompt tuning approach using only internal augmentation on raw dataset to better exploit known features. Specifically, AugPT employs self-supervised augmentation on unlabeled images in the training set, and introduces a novel gating mechanism based on consensus test, reusing the pre-trained prompt tuning backbone model to spontaneously filter noisy samples, further enhancing the quality of augmented views. Extensive experiments validate that AugPT simultaneously enhances model performance and generalization capability without using appended external knowledge. The code of AugPT is available at: https://github.com/JREion/AugPT .

Related papers

• VIP-COP: Context Optimization for Tabular Foundation Models [23.854424850102273]
  VIP-COP estimates the Value of Importance for Prediction of training examples and features for hard Context OPtimization for TFMs.<n>Its explicit selection mechanism suppresses noise and isolates influential data, enabling the model to also benefit from data augmentation.<n> VIP-COP is (i) fast, boosting performance often within minutes of optimization; (ii) budget-aware and any-time, improving with additional test-time compute; (iii) model-aware yet fully black-box, requiring no access to model internals; (iv) interpretable, identifying discrete Very Important Predictors'' (sample
  arXiv  Detail & Related papers  (2026-05-13T02:28:31Z)
• On Improving Deep Active Learning with Formal Verification [3.925313161884993]
  We investigate how augmenting the training data with adversarial inputs that violate robustness constraints can improve DAL performance.<n>We show that adversarial examples generated via formal verification contribute substantially more than those produced by standard, gradient-based attacks.
  arXiv  Detail & Related papers  (2025-12-16T08:01:42Z)
• MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models [37.85176585188362]
  We propose Model-Agnostic Optimization (MAO) for prompt tuning.<n>We introduce a Data-Driven Enhancement framework to optimize the distribution of the initial data.<n>We incorporate an Alterable Regularization module to boost the task-specific feature processing pipeline.
  arXiv  Detail & Related papers  (2025-03-23T17:59:33Z)
• Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models [54.385486006684495]
  LaPael is a latent-level paraphrasing method that applies input-dependent noise to early Large Language Models layers. Our experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches.
  arXiv  Detail & Related papers  (2024-11-01T15:47:05Z)
• DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception [78.26734070960886]
  Current perceptive models heavily depend on resource-intensive datasets. We introduce perception-aware loss (P.A. loss) through segmentation, improving both quality and controllability. Our method customizes data augmentation by extracting and utilizing perception-aware attribute (P.A. Attr) during generation.
  arXiv  Detail & Related papers  (2024-03-20T04:58:03Z)
• Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification [3.129187821625805]
  Auxiliary Fourier-basis Augmentation (AFA) is a technique targeting augmentation in the frequency domain and filling the augmentation gap left by visual augmentations. Our results show that AFA benefits the robustness of models against common corruptions, OOD generalization, and consistency of performance of models against increasing perturbations, with negligible deficit to the standard performance of models.
  arXiv  Detail & Related papers  (2024-03-04T11:30:02Z)
• Graph Masked Autoencoder for Sequential Recommendation [10.319298705782058]
  We propose a Graph Masked AutoEncoder-enhanced sequential Recommender system (MAERec) that adaptively and dynamically distills global item transitional information for self-supervised augmentation. Our method significantly outperforms state-of-the-art baseline models and can learn more accurate representations against data noise and sparsity.
  arXiv  Detail & Related papers  (2023-05-08T10:57:56Z)
• Effectiveness of Data Augmentation for Parameter Efficient Tuning with Limited Data [30.869230680173825]
  We show that data augmentation can be used to boost the performance of P-tuning and LoRA models. We show how P-tuning presents a more limited ability to separate the sentence embeddings from different classes of augmented data.
  arXiv  Detail & Related papers  (2023-03-05T04:12:17Z)
• On-the-fly Denoising for Data Augmentation in Natural Language Understanding [101.46848743193358]
  We propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data. Our method can be applied to general augmentation techniques and consistently improve the performance on both text classification and question-answering tasks.
  arXiv  Detail & Related papers  (2022-12-20T18:58:33Z)
• Cross-Modal Generative Augmentation for Visual Question Answering [34.9601948665926]
  This paper introduces a generative model for data augmentation by leveraging the correlations among multiple modalities. The proposed model is able to quantify the confidence of augmented data by its generative probability, and can be jointly updated with a downstream pipeline.
  arXiv  Detail & Related papers  (2021-05-11T04:51:26Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)
• Simple is Better! Lightweight Data Augmentation for Low Resource Slot Filling and Intent Classification [3.2996723916635267]
  We show that lightweight augmentation, a set of augmentation methods involving word span and sentence level operations, alleviates data scarcity problems. Experiments on limited data settings show that lightweight augmentation yields significant performance improvement on slot filling. Lightweight augmentation is also beneficial when combined with pre-trained LM-based models, as it improves BERT-based joint intent and slot filling models.
  arXiv  Detail & Related papers  (2020-09-08T12:39:47Z)
• Generative Data Augmentation for Commonsense Reasoning [75.26876609249197]
  G-DAUGC is a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. G-DAUGC consistently outperforms existing data augmentation methods based on back-translation. Our analysis demonstrates that G-DAUGC produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.
  arXiv  Detail & Related papers  (2020-04-24T06:12:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.