What Makes Good Examples for Visual In-Context Learning?
- URL: http://arxiv.org/abs/2301.13670v2
- Date: Wed, 1 Feb 2023 08:38:14 GMT
- Title: What Makes Good Examples for Visual In-Context Learning?
- Authors: Yuanhan Zhang, Kaiyang Zhou, Ziwei Liu
- Abstract summary: We focus on an emergent ability in large vision models, known as in-context learning, which allows inference on unseen tasks by conditioning on in-context examples.
We propose a prompt retrieval framework to automate the selection of in-context examples.
Specifically, we present (1) an unsupervised prompt retrieval method based on nearest example search using an off-the-shelf model, and (2) a supervised prompt retrieval method, which trains a neural network to choose examples that directly maximize in-context learning performance.
- Score: 38.68910532066619
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Large-scale models trained on broad data have recently become the mainstream
architecture in computer vision due to their strong generalization performance.
In this paper, the main focus is on an emergent ability in large vision models,
known as in-context learning, which allows inference on unseen tasks by
conditioning on in-context examples (a.k.a.~prompt) without updating the model
parameters. This concept has been well-known in natural language processing but
has only been studied very recently for large vision models. We for the first
time provide a comprehensive investigation on the impact of in-context examples
in computer vision, and find that the performance is highly sensitive to the
choice of in-context examples. To overcome the problem, we propose a prompt
retrieval framework to automate the selection of in-context examples.
Specifically, we present (1) an unsupervised prompt retrieval method based on
nearest example search using an off-the-shelf model, and (2) a supervised
prompt retrieval method, which trains a neural network to choose examples that
directly maximize in-context learning performance. The results demonstrate that
our methods can bring non-trivial improvements to visual in-context learning in
comparison to the commonly-used random selection.
Related papers
- SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
We propose a framework to enable in-context learning in large language models.
A meta-model can learn on self-supervised prompts consisting of tailored demonstrations.
Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
arXiv Detail & Related papers (2023-07-15T08:33:08Z) - Retrieval-Enhanced Visual Prompt Learning for Few-shot Classification [9.843214426749764]
Retrieval-enhanced Prompt learning (RePrompt)
We propose Retrieval-enhanced Prompt learning (RePrompt), which introduces retrieval mechanisms to cache the knowledge representations from downstream tasks.
Our experiments over 15 vision datasets, including 11 downstream tasks with few-shot setting and 4 domain generalization benchmarks, demonstrate that RePrompt achieves considerably improved performance.
arXiv Detail & Related papers (2023-06-04T03:06:37Z) - RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning.
We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
arXiv Detail & Related papers (2023-05-23T20:15:56Z) - Exploring Effective Factors for Improving Visual In-Context Learning [56.14208975380607]
In-Context Learning (ICL) is to understand a new task via a few demonstrations (aka. prompt) and predict new inputs without tuning the models.
This paper shows that prompt selection and prompt fusion are two major factors that have a direct impact on the inference performance of visual context learning.
We propose a simple framework prompt-SelF for visual in-context learning.
arXiv Detail & Related papers (2023-04-10T17:59:04Z) - Fairness-guided Few-shot Prompting for Large Language Models [93.05624064699965]
In-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats.
We introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes.
We propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning.
arXiv Detail & Related papers (2023-03-23T12:28:25Z) - Reordering Examples Helps during Priming-based Few-Shot Learning [6.579039107070663]
We show that PERO can learn to generalize efficiently using as few as 10 examples.
We demonstrate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval.
arXiv Detail & Related papers (2021-06-03T11:02:36Z) - Learning What Makes a Difference from Counterfactual Examples and
Gradient Supervision [57.14468881854616]
We propose an auxiliary training objective that improves the generalization capabilities of neural networks.
We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task.
Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
arXiv Detail & Related papers (2020-04-20T02:47:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.