Prompt Distribution Learning

• URL: http://arxiv.org/abs/2205.03340v1
• Date: Fri, 6 May 2022 16:22:36 GMT
• Title: Prompt Distribution Learning
• Authors: Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, Xinmei Tian
• Abstract summary: We present prompt distribution learning for adapting a pre-trained vision-language model to address downstream recognition tasks. Our method not only learns low-bias prompts from a few samples but also captures the distribution of diverse prompts to handle the varying visual representations. This prompt distribution learning is realized by an efficient approach that learns the output embeddings of prompts instead of the input embeddings.
• Score: 46.46876752213575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present prompt distribution learning for effectively adapting a pre-trained vision-language model to address downstream recognition tasks. Our method not only learns low-bias prompts from a few samples but also captures the distribution of diverse prompts to handle the varying visual representations. In this way, we provide high-quality task-related content for facilitating recognition. This prompt distribution learning is realized by an efficient approach that learns the output embeddings of prompts instead of the input embeddings. Thus, we can employ a Gaussian distribution to model them effectively and derive a surrogate loss for efficient training. Extensive experiments on 12 datasets demonstrate that our method consistently and significantly outperforms existing methods. For example, with 1 sample per category, it relatively improves the average result by 9.1% compared to human-crafted prompts.

Related papers

• Adaptive teachers for amortized samplers [76.88721198565861]
  Amortized inference is the task of training a parametric model, such as a neural network, to approximate a distribution with a given unnormalized density where exact sampling is intractable. Off-policy RL training facilitates the discovery of diverse, high-reward candidates, but existing methods still face challenges in efficient exploration. We propose an adaptive training distribution (the Teacher) to guide the training of the primary amortized sampler (the Student) by prioritizing high-loss regions.
  arXiv  Detail & Related papers  (2024-10-02T11:33:13Z)
• When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective [57.05315507519704]
  We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of visual prompting and linear probing. Our measure attains up to a 100-fold reduction in run time compared to full training, while achieving prediction accuracies up to 91%.
  arXiv  Detail & Related papers  (2024-09-03T12:03:45Z)
• Self-Knowledge Distillation for Learning Ambiguity [11.755814660833549]
  Recent language models often over-confidently predict a single label without consideration for its correctness. We propose a novel self-knowledge distillation method that enables models to learn label distributions more accurately. We validate our method on diverse NLU benchmark datasets and the experimental results demonstrate its effectiveness in producing better label distributions.
  arXiv  Detail & Related papers  (2024-06-14T05:11:32Z)
• Simple and effective data augmentation for compositional generalization [64.00420578048855]
  We show that data augmentation methods that sample MRs and backtranslate them can be effective for compositional generalization. Remarkably, sampling from a uniform distribution performs almost as well as sampling from the test distribution.
  arXiv  Detail & Related papers  (2024-01-18T09:13:59Z)
• POODLE: Improving Few-shot Learning via Penalizing Out-of-Distribution Samples [19.311470287767385]
  We propose to use out-of-distribution samples, i.e., unlabeled samples coming from outside the target classes, to improve few-shot learning. Our approach is simple to implement, agnostic to feature extractors, lightweight without any additional cost for pre-training, and applicable to both inductive and transductive settings.
  arXiv  Detail & Related papers  (2022-06-08T18:59:21Z)
• Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning [3.1153758106426603]
  Few-shot classification is a challenging problem due to the uncertainty caused by using few labelled samples. We propose a novel transfer-based method which aims at processing the feature vectors so that they become closer to Gaussian-like distributions. In the case of transductive few-shot learning where unlabelled test samples are available during training, we also introduce an optimal-transport inspired algorithm to boost even further the achieved performance.
  arXiv  Detail & Related papers  (2021-10-18T16:29:17Z)
• Contrastive Learning for Fair Representations [50.95604482330149]
  Trained classification models can unintentionally lead to biased representations and predictions. Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and difficult to optimise. We propose a method for mitigating bias by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations.
  arXiv  Detail & Related papers  (2021-09-22T10:47:51Z)
• An Effective Baseline for Robustness to Distributional Shift [5.627346969563955]
  Refraining from confidently predicting when faced with categories of inputs different from those seen during training is an important requirement for the safe deployment of deep learning systems. We present a simple, but highly effective approach to deal with out-of-distribution detection that uses the principle of abstention.
  arXiv  Detail & Related papers  (2021-05-15T00:46:11Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.