It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

• URL: http://arxiv.org/abs/2607.18232v2
• Date: Wed, 22 Jul 2026 15:51:58 GMT
• Title: It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
• Abstract summary: We introduce a typology to evaluate how different expressions of belief (EoBs) affect whether models follow context versus prior knowledge.<n>We generate controlled EoB-query pairs that isolate the effect of linguistic variation.<n>Our work reveals systematic patterns in how linguistic framing affects context integration, with implications for prompt engineering and model robustness.
• Score: 14.724790683440375
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Users frequently express their beliefs to large language models (LLMs). In some situations, the LLM should accept these contextual beliefs as true. In others, they should stick to their prior knowledge. Notably, users' expressions of belief (EoBs) can take linguistically diverse forms - using presuppositions, evidential and certainty markers, or varied tones - each of which may have a different persuasiveness over the LLMs. We introduce a typology to systematically evaluate how different EoBs affect whether models follow context versus prior knowledge. The typology is grounded in four linguistically motivated dimensions: form, evidentiality, epistemic stance, and tone, spanning 17 fine-grained types. By pairing these EoBs with world knowledge facts, we generate controlled EoB-query pairs that isolate the effect of linguistic variation. Using this benchmark, we evaluate 16 LLMs that differ in architecture (Llama3, Qwen3, Gemma3), scale (1B-30B parameters), and training stages (base vs instruct). We identify meaningful variations in response behavior across these axes, e.g., that bigger models and instruction models tend to be less context-following than smaller models and base models. We further identify specific EoBs that statistically significantly persuade LMs more consistently than others. Our work reveals systematic patterns in how linguistic framing affects LLM context integration, with implications for prompt engineering and model robustness.

Related papers

• Words as Difference Makers: How Large Language Models Determine Causal Structure in Text [0.0]
  I argue that large language models (LLMs) employ a specific inductive approach based on a difference-making logic.<n>I analyze specific architectural features of LLMs to determine their roles in variational induction.
  arXiv  Detail & Related papers  (2026-06-21T10:40:47Z)
• Epistemic Diversity and Knowledge Collapse in Large Language Models [42.81169404715417]
  Large language models (LLMs) tend to generate lexically, semantically, and stylistically homogenous texts.<n>This poses a risk of knowledge collapse, where homogenous LLMs mediate a shrinking in the range of accessible information over time.<n>We present a new methodology to measure epistemic diversity, i.e., variation in real-world claims in LLM outputs.
  arXiv  Detail & Related papers  (2025-10-05T14:29:15Z)
• Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? [81.49470136653665]
  We evaluate the robustness and expressiveness of value representations across three widely used probing strategies.<n>We show that the demographic context has little effect on the free-text generation, and the models' values only weakly correlate with their preference for value-based actions.
  arXiv  Detail & Related papers  (2025-07-17T18:56:41Z)
• A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Bias [1.7109513360384465]
  Large Language Models (LLMs) represent a major step toward artificial general intelligence.<n>Questions remain about their output similarity, variability, and ethical implications.
  arXiv  Detail & Related papers  (2025-05-14T01:21:46Z)
• Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English [66.97110551643722]
  We investigate dialectal disparities in Large Language Models (LLMs) reasoning tasks.<n>We find that LLMs produce less accurate responses and simpler reasoning chains and explanations for AAE inputs.<n>These findings highlight systematic differences in how LLMs process and reason about different language varieties.
  arXiv  Detail & Related papers  (2025-03-06T05:15:34Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Do LLMs write like humans? Variation in grammatical and rhetorical styles [0.6303112417588329]
  Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems.<n>As they have advanced, it has become difficult to distinguish their output from human-written text.
  arXiv  Detail & Related papers  (2024-10-21T15:35:44Z)
• LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements [59.71218039095155]
  Task of reading comprehension (RC) provides a primary means to assess language models' natural language understanding (NLU) capabilities.<n>If the context aligns with the models' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from internal information.<n>To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities.
  arXiv  Detail & Related papers  (2024-04-09T13:08:56Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs [26.118193748582197]
  We evaluate four categories of widely used state-of-the-art models. We find that, despite only evaluating on utterances that require a binary inference, models in three of these categories perform close to random. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models.
  arXiv  Detail & Related papers  (2022-10-26T19:04:23Z)
• Probing Pretrained Language Models for Lexical Semantics [76.73599166020307]
  We present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks. Our results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks.
  arXiv  Detail & Related papers  (2020-10-12T14:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.