Test-Time Adaptation via Many-Shot Prompting: Benefits, Limits, and Pitfalls

• URL: http://arxiv.org/abs/2603.05829v2
• Date: Thu, 12 Mar 2026 23:57:58 GMT
• Title: Test-Time Adaptation via Many-Shot Prompting: Benefits, Limits, and Pitfalls
• Abstract summary: Test-time adaptation enables large language models to modify their behavior at inference without updating model parameters.<n>Many-shot prompting is a common approach, where large numbers of in-context learning examples are injected as an input-space test-time update.<n>We present an empirical study of many-shot prompting across tasks and model backbones, analyzing how performance varies with update magnitude, example ordering, and selection policy.<n>We find that many-shot prompting is effective for structured tasks where demonstrations provide high information gain, but is highly sensitive to selection strategy and often shows limited benefits for open-ended generation tasks.
• Score: 6.538221912304088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time adaptation enables large language models (LLMs) to modify their behavior at inference without updating model parameters. A common approach is many-shot prompting, where large numbers of in-context learning (ICL) examples are injected as an input-space test-time update. Although performance can improve as more demonstrations are added, the reliability and limits of this update mechanism remain poorly understood, particularly for open-source models. We present an empirical study of many-shot prompting across tasks and model backbones, analyzing how performance varies with update magnitude, example ordering, and selection policy. We further study Dynamic and Reinforced ICL as alternative test-time update strategies that control which information is injected and how it constrains model behavior. We find that many-shot prompting is effective for structured tasks where demonstrations provide high information gain, but is highly sensitive to selection strategy and often shows limited benefits for open-ended generation tasks. Overall, we characterize the practical limits of prompt-based test-time adaptation and outline when input-space updates are beneficial versus harmful.

Related papers

• Test-Time Learning with an Evolving Library [76.87197412229318]
  EvoLib is a test-time learning framework that enables large language models to accumulate, reuse, and evolve knowledge.<n>Instead of adapting model parameters, our approach maintains a shared library of knowledge abstractions.
  arXiv  Detail & Related papers  (2026-05-14T07:18:12Z)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models [49.04912820721943]
  Reinforcement learning (RL) finetuning has become a key technique for enhancing the reasoning abilities of large language models (LLMs)<n>Recent advances underscore the importance of online prompt selection methods, which typically concentrate training on partially solved or moderately challenging examples.<n>This work proposes Dynamics-Predictive Sampling (DPS), which online predicts and selects informative prompts by inferring their learning dynamics prior to costly rollouts.
  arXiv  Detail & Related papers  (2026-03-11T15:31:14Z)
• Efficient Test-Time Scaling for Small Vision-Language Models [14.654047034885288]
  Small Vision-Language Models (VLMs) provide a computationally efficient alternative to larger models.<n>Existing methods are typically computationally demanding, contradicting the resource-efficient design goals of small models.<n>We propose two novel and efficient test-time scaling strategies that leverage the model-internal features rather than external supervision.
  arXiv  Detail & Related papers  (2025-10-03T23:49:06Z)
• TAPS : Frustratingly Simple Test Time Active Learning for VLMs [0.0]
  Test-Time Optimization enables models to adapt to new data during inference by updating parameters on-the-fly.<n>We propose a novel Test-Time Active Learning framework that adaptively queries uncertain samples and updates prompts dynamically.<n>Our framework provides a practical and effective solution for real-world deployment in safety-critical applications such as autonomous systems and medical diagnostics.
  arXiv  Detail & Related papers  (2025-07-26T18:04:49Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• Space Rotation with Basis Transformation for Training-free Test-Time Adaptation [25.408849667998993]
  We propose a training-free feature space rotation with basis transformation for test-time adaptation.<n>By leveraging the inherent distinctions among classes, we reconstruct the original feature space and map it to a new representation.<n>Our method outperforms state-of-the-art techniques in terms of both performance and efficiency.
  arXiv  Detail & Related papers  (2025-02-27T10:15:34Z)
• Realistic Test-Time Adaptation of Vision-Language Models [23.972884634610413]
  Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance.<n>Previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution.<n>Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework.
  arXiv  Detail & Related papers  (2025-01-07T12:17:25Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Efficient Open Set Single Image Test Time Adaptation of Vision Language Models [15.621092104244003]
  Adapting models to dynamic, real-world environments is a critical challenge in deep learning.<n>We propose ROSITA, a novel framework that leverages dynamically updated feature banks to identify reliable test samples.<n>Our approach effectively adapts models to domain shifts for known classes while rejecting unfamiliar samples.
  arXiv  Detail & Related papers  (2024-06-01T16:21:42Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters.<n>We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL.<n>We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.