Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, LLaMA

• URL: http://arxiv.org/abs/2512.12812v1
• Date: Sun, 14 Dec 2025 19:25:20 GMT
• Title: Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, LLaMA
• Authors: Hanyu Cai, Binqi Shen, Lier Jin, Lan Hu, Xiaojing Fan,
• Abstract summary: This work proposes a systematic evaluation framework to examine how interaction tone affects model accuracy.<n>We apply this framework to three recently released and widely available large language models: GPT-4o mini (OpenAI), Gemini 2.0 Flash (Google DeepMind), and Llama 4 Scout (Meta)<n>Our results show that tone sensitivity is both model-dependent and domain-specific. Neutral or Very Friendly prompts generally yield higher accuracy than Very Rude prompts, but statistically significant effects appear only in a subset of Humanities tasks.
• Score: 0.6263481844384227
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt engineering has emerged as a critical factor influencing large language model (LLM) performance, yet the impact of pragmatic elements such as linguistic tone and politeness remains underexplored, particularly across different model families. In this work, we propose a systematic evaluation framework to examine how interaction tone affects model accuracy and apply it to three recently released and widely available LLMs: GPT-4o mini (OpenAI), Gemini 2.0 Flash (Google DeepMind), and Llama 4 Scout (Meta). Using the MMMLU benchmark, we evaluate model performance under Very Friendly, Neutral, and Very Rude prompt variants across six tasks spanning STEM and Humanities domains, and analyze pairwise accuracy differences with statistical significance testing. Our results show that tone sensitivity is both model-dependent and domain-specific. Neutral or Very Friendly prompts generally yield higher accuracy than Very Rude prompts, but statistically significant effects appear only in a subset of Humanities tasks, where rude tone reduces accuracy for GPT and Llama, while Gemini remains comparatively tone-insensitive. When performance is aggregated across tasks within each domain, tone effects diminish and largely lose statistical significance. Compared with earlier researches, these findings suggest that dataset scale and coverage materially influence the detection of tone effects. Overall, our study indicates that while interaction tone can matter in specific interpretive settings, modern LLMs are broadly robust to tonal variation in typical mixed-domain use, providing practical guidance for prompt design and model selection in real-world deployments.

Related papers

• Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interaction [1.3511057160494195]
  Leader-follower interaction is an important paradigm in human-robot interaction (HRI)<n>Small language models (SLMs) offer a potential alternative, but their effectiveness for role classification in HRI has not been systematically evaluated.
  arXiv  Detail & Related papers  (2026-02-26T18:20:26Z)
• LVLM-Aided Alignment of Task-Specific Vision Models [49.96265491629163]
  Small task-specific vision models are crucial in high-stakes domains.<n>We introduce a novel and efficient method for aligning small task-specific vision models with human domain knowledge.<n>Our method demonstrates substantial improvement in aligning model behavior with human specifications.
  arXiv  Detail & Related papers  (2025-12-26T11:11:25Z)
• Emergent Bayesian Behaviour and Optimal Cue Combination in LLMs [6.415869990358189]
  Large language models (LLMs) excel at explicit reasoning, but their implicit computational strategies remain underexplored.<n>We ask whether LLMs exhibit similar behaviour and perform optimal multimodal integration without explicit training or instruction.<n>We introduce a behavioural benchmark - BayesBench: four magnitude estimation tasks over text and image.<n>We measure performance, behaviour and efficiency in multimodal cue-combination.
  arXiv  Detail & Related papers  (2025-12-02T12:51:30Z)
• What Works for 'Lost-in-the-Middle' in LLMs? A Study on GM-Extract and Mitigations [1.2879523047871226]
  GM-Extract is a novel benchmark dataset meticulously designed to evaluate LLM performance on retrieval of control variables.<n>We conduct a systematic evaluation of 7-8B parameter models on two multi-document tasks (key-value extraction and question-answering)<n>While a distinct U-shaped curve was not consistently observed, our analysis reveals a clear pattern of performance across models.
  arXiv  Detail & Related papers  (2025-11-17T20:50:50Z)
• Grounded Test-Time Adaptation for LLM Agents [75.62784644919803]
  Large language model (LLM)-based agents struggle to generalize to novel and complex environments.<n>We propose two strategies for adapting LLM agents by leveraging environment-specific information available during deployment.
  arXiv  Detail & Related papers  (2025-11-06T22:24:35Z)
• Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning [19.313795358097483]
  In-context learning (ICL) is a critical emerging capability of large language models (LLMs)<n>This paper investigates an unexplored new positional bias of ICL for the first time.<n>We observe that the predictions and accuracy can drift drastically when the positions of demos, the system prompt, and the user message are varied.
  arXiv  Detail & Related papers  (2025-07-30T17:59:46Z)
• Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? [81.49470136653665]
  We evaluate the robustness and expressiveness of value representations across three widely used probing strategies.<n>We show that the demographic context has little effect on the free-text generation, and the models' values only weakly correlate with their preference for value-based actions.
  arXiv  Detail & Related papers  (2025-07-17T18:56:41Z)
• Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation [54.3628937181904]
  Internal world models (WMs) enable agents to understand the world's state and predict transitions.<n>Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs.
  arXiv  Detail & Related papers  (2025-06-27T03:24:29Z)
• Evaluating the Sensitivity of LLMs to Prior Context [2.377922603550519]
  Large language models (LLMs) are increasingly deployed in multi-turn dialogue and other sustained interactive scenarios.<n>We introduce a novel set of benchmarks that vary the volume and nature of prior context to measure sensitivity to contextual variations.<n>Our findings reveal that LLM performance on multiple-choice questions can degrade dramatically in multi-turn interactions.
  arXiv  Detail & Related papers  (2025-05-29T16:09:32Z)
• If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs [55.8331366739144]
  We introduce LIFESTATE-BENCH, a benchmark designed to assess lifelong learning in large language models (LLMs)<n>Our fact checking evaluation probes models' self-awareness, episodic memory retrieval, and relationship tracking, across both parametric and non-parametric approaches.
  arXiv  Detail & Related papers  (2025-03-30T16:50:57Z)
• Fact-checking with Generative AI: A Systematic Cross-Topic Examination of LLMs Capacity to Detect Veracity of Political Information [0.0]
  The purpose of this study is to assess how large language models (LLMs) can be used for fact-checking.<n>We use AI auditing methodology that systematically evaluates performance of five LLMs.<n>The results indicate that models are better at identifying false statements, especially on sensitive topics.
  arXiv  Detail & Related papers  (2025-03-11T13:06:40Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.