Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priors

• URL: http://arxiv.org/abs/2510.08413v1
• Date: Thu, 09 Oct 2025 16:32:46 GMT
• Title: Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priors
• Authors: David Madras, Joshua Safyan, Qiuyi, Zhang,
• Abstract summary: We argue that such widespread success can be more fully explained through more carefully considering data- or distribution-dependent perplexity.<n>We derive novel generalization bounds that are non-vacuous for data-scarce prompt optimization via more useful priors.
• Score: 26.84417698402442
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many prompt engineering techniques have been successful in practice, even when optimizing over a large prompt space with with a small amount of task-specific data. Recent work has partially explained this success by showing generalization bounds which apply PAC-Bayes theory to the discrete prompt space, but they are non-vacuous only in data-rich scenarios. We argue that such widespread success can be more fully explained through more carefully considering data- or distribution-dependent perplexity, which acts as an effective prior and steers the optimization towards prompts that are more ``natural'' for the task at hand. We derive novel generalization bounds that are non-vacuous for data-scarce prompt optimization via more useful priors, formally analyzing how perplexity regularization tightens these bounds by limiting exploration. Empirically, we explore both the bounds' effectiveness and the practical benefits of perplexity regularization in improving prompt generalization.

Related papers

• DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization [10.083326281775939]
  Large language models (LLMs) have achieved impressive performance in text summarization.<n>Fine-tuning can improve summarization quality, but it typically relies on costly and scarce high-quality labeled data.<n>We explore continual pre-training as a scalable, self-supervised approach to adapt LLMs for downstream summarization tasks.
  arXiv  Detail & Related papers  (2025-10-07T12:26:19Z)
• Boosting Private Domain Understanding of Efficient MLLMs: A Tuning-free, Adaptive, Universal Prompt Optimization Framework [60.26747209785186]
  multimodal large language models (EMLLMs) reduce model size and computational costs and are often deployed on resource-constrained devices.<n>Existing open-sourceLMs rarely have access to private domain-specific data during the pre-training process.<n>We propose a tuntextbfunderlineIng-free, atextbfunderlineDaptivtextbfunderlineE, universtextbfunderlineAL textbfunderlinePrompt Optimization Framework.
  arXiv  Detail & Related papers  (2024-12-27T15:21:17Z)
• A PAC-Bayesian Link Between Generalisation and Flat Minima [32.96252933767587]
  We provide novel generalisation bounds involving gradient terms.<n>We highlight the positive influence of flat minima on generalisation performance, involving directly the benefits of the optimisation phase.
  arXiv  Detail & Related papers  (2024-02-13T15:03:02Z)
• A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models [19.17722702457403]
  We show that state-of-the-artETL approaches exhibit strong performance only in narrowly-defined experimental setups. We propose a CLass-Adaptive linear Probe (CLAP) objective, whose balancing term is optimized via an adaptation of the general Augmented Lagrangian method.
  arXiv  Detail & Related papers  (2023-12-20T02:58:25Z)
• Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
  We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
  arXiv  Detail & Related papers  (2023-11-03T03:18:40Z)
• Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning [83.41487567765871]
  Skipper is a model-based reinforcement learning framework. It automatically generalizes the task given into smaller, more manageable subtasks. It enables sparse decision-making and focused abstractions on the relevant parts of the environment.
  arXiv  Detail & Related papers  (2023-09-30T02:25:18Z)
• PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization [48.26492774959634]
  We develop a compression approach based on quantizing neural network parameters in a linear subspace. We find large models can be compressed to a much greater extent than previously known, encapsulating Occam's razor.
  arXiv  Detail & Related papers  (2022-11-24T13:50:16Z)
• Instance-Dependent Generalization Bounds via Optimal Transport [51.71650746285469]
  Existing generalization bounds fail to explain crucial factors that drive the generalization of modern neural networks. We derive instance-dependent generalization bounds that depend on the local Lipschitz regularity of the learned prediction function in the data space. We empirically analyze our generalization bounds for neural networks, showing that the bound values are meaningful and capture the effect of popular regularization methods during training.
  arXiv  Detail & Related papers  (2022-11-02T16:39:42Z)
• Efficient Contextual Preferential Bayesian Optimization with Historical Examples [0.5249805590164902]
  We propose an offline, interpretable utility learning method that uses expert knowledge, historical examples, and coarse information about the utility space to reduce sample requirements.<n>Our method outperforms standard Gaussian processes and BOPE across four domains, showing strong performance even with biased samples, as encountered in the real-world, and limited expert input.
  arXiv  Detail & Related papers  (2022-08-22T13:26:03Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.