Understanding the Prompt Sensitivity

• URL: http://arxiv.org/abs/2604.18389v1
• Date: Mon, 20 Apr 2026 15:13:33 GMT
• Title: Understanding the Prompt Sensitivity
• Abstract summary: Prompt sensitivity refers to how strongly the output of a large language model depends on the exact wording of its input prompt.<n>We analyze the relationship between meaning-preserving prompts, their gradients, and the log probabilities of the model's next token.<n>We show that LLMs do not internally cluster similar inputs like smaller neural networks do, but instead disperse them.
• Score: 24.53996114318076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt sensitivity, which refers to how strongly the output of a large language model (LLM) depends on the exact wording of its input prompt, raises concerns among users about the LLM's stability and reliability. In this work, we consider LLMs as multivariate functions and perform a first-order Taylor expansion, thereby analyzing the relationship between meaning-preserving prompts, their gradients, and the log probabilities of the model's next token. We derive an upper bound on the difference between log probabilities using the Cauchy-Schwarz inequality. We show that LLMs do not internally cluster similar inputs like smaller neural networks do, but instead disperse them. This dispersing behavior leads to an excessively high upper bound on the difference of log probabilities between two meaning-preserving prompts, making it difficult to effectively reduce to 0. In our analysis, we also show which types of meaning-preserving prompt variants are more likely to introduce prompt sensitivity risks in LLMs. In addition, we demonstrate that the upper bound is strongly correlated with an existing prompt sensitivity metric, PromptSensiScore. Moreover, by analyzing the logit variance, we find that prompt templates typically exert a greater influence on logits than the questions themselves. Overall, our results provide a general interpretation for why current LLMs can be highly sensitive to prompts with the same meaning, offering crucial evidence for understanding the prompt sensitivity of LLMs. Code for experiments is available at https://github.com/ku-nlp/Understanding_the_Prompt_Sensitivity.

Related papers

• Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions [10.595427068095397]
  Even subtle and semantically irrelevant changes in prompts can cause dramatic fluctuations in performance.<n>We propose an Interaction-based Prompt Sensitivity metric by quantifying changes in interactions when we introduce subtle changes to prompts.
  arXiv  Detail & Related papers  (2026-08-19T04:45:59Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs [100.02824137397464]
  We investigate how Large Language Models adapt their internal representations when encountering inputs of increasing difficulty.<n>We reveal a consistent and quantifiable phenomenon: as task difficulty increases, the last hidden states of LLMs become substantially sparser.<n>This sparsity--difficulty relation is observable across diverse models and domains.
  arXiv  Detail & Related papers  (2026-03-03T18:48:15Z)
• Revisiting Prompt Sensitivity in Large Language Models for Text Classification: The Role of Prompt Underspecification [3.2059646106414967]
  Large language models (LLMs) are widely used as zero-shot and few-shot classifiers.<n>We study and compare the sensitivity of underspecified prompts and prompts that provide specific instructions.<n>We find that underspecified prompts exhibit higher performance variance and lower logit values for relevant tokens, while instruction-prompts suffer less from such problems.
  arXiv  Detail & Related papers  (2026-02-04T07:59:28Z)
• Visualizing token importance for black-box language models [48.747801442240565]
  We consider the problem of auditing black-box large language models (LLMs) to ensure they behave reliably when deployed in production settings.<n>We propose Distribution-Based Sensitivity Analysis (DBSA) to evaluate the sensitivity of the output of a language model for each input token.
  arXiv  Detail & Related papers  (2025-12-12T14:01:43Z)
• Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank [71.09032766271493]
  Large language models (LLMs) are prone to errors and hallucinations.<n>How to check their outputs effectively and efficiently has become a critical problem in their applications.
  arXiv  Detail & Related papers  (2025-10-28T11:01:10Z)
• Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models [39.05891782057066]
  We study prompt sensitivity in large language models (LLMs)<n>We show that sampling across the semantic concept space'' with paraphrasing perturbations improves uncertainty calibration without compromising accuracy.
  arXiv  Detail & Related papers  (2025-10-19T22:28:57Z)
• Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs [34.51801559719707]
  High prompt sensitivity has been widely accepted as a core limitation of large language models.<n>This work asks: Is the widely reported high prompt sensitivity truly an inherent weakness of LLMs, or is it largely an artifact of evaluation processes?<n>We find that much of the prompt sensitivity stems from evaluation methods, including log-likelihood scoring and rigid answer matching.
  arXiv  Detail & Related papers  (2025-09-01T21:38:28Z)
• ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs [72.13489820420726]
  ProSA is a framework designed to evaluate and comprehend prompt sensitivity in large language models. Our study uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness.
  arXiv  Detail & Related papers  (2024-10-16T09:38:13Z)
• How are Prompts Different in Terms of Sensitivity? [50.67313477651395]
  We present a comprehensive prompt analysis based on the sensitivity of a function. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output. We introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding.
  arXiv  Detail & Related papers  (2023-11-13T10:52:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.