Measuring LLM Sycophancy under Sustained Multi-Turn Pressure

• URL: http://arxiv.org/abs/2609.09090v1
• Date: Tue, 08 Sep 2026 17:35:04 GMT
• Title: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
• Abstract summary: Large language models (LLMs) may abandon correct positions when users push back, exhibiting a failure mode known as sycophancy.<n>We introduce SPINE, a benchmark in which an LLM proxy plays a persistent but mistaken user and adaptively challenges a target model for up to 25 turns.
• Score: 16.810854941543926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) may abandon correct positions when users push back, exhibiting a failure mode known as sycophancy. Existing evaluations typically use short, pre-specified conversations and may therefore miss failures that emerge under sustained, adaptive disagreement. We introduce SPINE, a benchmark in which an LLM proxy plays a persistent but mistaken user and adaptively challenges a target model for up to 25 turns. We evaluate four production systems and three Olmo3-7b variants on 100 false-presupposition and 100 unethical-query items. Our experimental results show that collapse rates increase with conversation length for every model, short-horizon protocols underestimate sycophancy and resistance under sustained pressure remains unreliable across current models. By analyzing models with accessible reasoning traces, we surprisingly found that the correct position often remains represented in a reasoning trace when the response concedes, suggesting that the model chooses to please a user and sycophancy is not due to lack of knowledge or ignorance. Ablations show that adaptive LLM proxy exposes more sycophantic collapse than pre-generated scripts. Among all tactics, emotional appeals is the most associated with inducing LLM sycophantic behavior. The code and data are released at https://anonymous.4open.science/r/SPINE

Related papers

• Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure [6.892220883660279]
  We introduce a benchmark and dataset for evaluating LMRM sycophancy when confronted with a wrong answer from a user.<n>We find that sycophancy is prevalent under pressure, with Statement eliciting the highest rates and Conviction the lowest.<n>We introduce a failure taxonomy separating reasoning-chain from answer-level sycophancy, and a complementary sentence-level taxonomy locating where in the chain drift first emerges.
  arXiv  Detail & Related papers  (2026-07-30T03:45:46Z)
• LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation [79.03892269184145]
  LivingArena is an automated, contamination-resistant evaluation framework.<n>Models take turns proposing questions, aiming to pose items that opponents cannot answer correctly.<n>Questioners are encouraged to actively identify and exploit opponents' knowledge boundaries.
  arXiv  Detail & Related papers  (2026-06-19T06:20:58Z)
• Verbalizing LLMs' assumptions to explain and control sycophancy [62.927670321859495]
  LLMs can be socially sycophantic, affirming users when they ask questions like "am I in the wrong?" rather than providing genuine assessment.<n>We present Verbalized Assumptions, a framework for eliciting these assumptions from LLMs.<n>Our work contributes a new understanding of assumptions as a mechanism for sycophancy.
  arXiv  Detail & Related papers  (2026-04-03T14:15:43Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models [2.1700203922407493]
  We propose a novel way to evaluate sycophancy of LLMs in a direct and neutral way.<n>A key novelty is the use of LLM-as-a-judge, evaluation of sycophancy as a zero-sum game in a bet setting.
  arXiv  Detail & Related papers  (2026-01-21T20:00:14Z)
• Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation [66.84286617519258]
  Large language models are transforming social science research by enabling the automation of labor-intensive tasks like data annotation and text analysis.<n>Such variation can introduce systematic biases and random errors, which propagate to downstream analyses and cause Type I (false positive), Type II (false negative), Type S (wrong sign), or Type M (exaggerated effect) errors.<n>We find that intentional LLM hacking is strikingly simple. By replicating 37 data annotation tasks from 21 published social science studies, we show that, with just a handful of prompt paraphrases, virtually anything can be presented as statistically significant.
  arXiv  Detail & Related papers  (2025-09-10T17:58:53Z)
• LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users [50.18141341939909]
  We describe a vulnerability in language models trained with user feedback.<n>A single user can persistently alter LM knowledge and behavior.<n>We show that this attack can be used to insert factual knowledge the model did not previously possess.
  arXiv  Detail & Related papers  (2025-07-03T17:55:40Z)
• Measuring Sycophancy of Language Models in Multi-turn Dialogues [33.875038658886986]
  We introduce SYCON Bench, a novel benchmark for evaluating sycophancy in multi-turn, free-form conversational settings.<n>Applying SYCON Bench to 17 Large Language Models across three real-world scenarios, we find that sycophancy remains a prevalent failure mode.
  arXiv  Detail & Related papers  (2025-05-28T14:05:46Z)
• ELEPHANT: Measuring and understanding social sycophancy in LLMs [31.88430788417527]
  We introduce social sycophancy, characterizing sycophancy as excessive preservation of a user's face.<n>Applying our benchmark to 11 models, we show that LLMs consistently exhibit high rates of social sycophancy.
  arXiv  Detail & Related papers  (2025-05-20T06:45:17Z)
• Simple synthetic data reduces sycophancy in large language models [88.4435858554904]
  We study the prevalence of sycophancy in language models. Sycophancy is where models tailor their responses to follow a human user's view even when that view is not objectively correct.
  arXiv  Detail & Related papers  (2023-08-07T23:48:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.