Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure

• URL: http://arxiv.org/abs/2608.28623v1
• Date: Thu, 30 Jul 2026 03:45:46 GMT
• Title: Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure
• Abstract summary: We introduce a benchmark and dataset for evaluating LMRM sycophancy when confronted with a wrong answer from a user.<n>We find that sycophancy is prevalent under pressure, with Statement eliciting the highest rates and Conviction the lowest.<n>We introduce a failure taxonomy separating reasoning-chain from answer-level sycophancy, and a complementary sentence-level taxonomy locating where in the chain drift first emerges.
• Score: 6.892220883660279
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large multimodal reasoning models (LMRMs) are getting increasingly capable, primarily through generating explicit chain-of-thought reasoning before answering. In language models it has been observed that this performance often comes with sycophancy, the tendency of a model to agree with the user over the evidence. However, for LMRMs no reliable method to measure sycophancy yet exists. We bridge this gap by introducing a benchmark and dataset for evaluating LMRM sycophancy when confronted with a wrong answer from a user. Our benchmark pairs four visually grounded datasets spanning mathematical, clinical, temporal, and demographic reasoning with five pressure conditions in single-turn and multi-turn settings. We evaluate sycophancy in the final answer as well as its emergence within the reasoning chain. We find that sycophancy is prevalent under pressure, with Statement pressure eliciting the highest rates and Conviction the lowest for all models except Mistral-Small-4, and under multi-turn pressure reasoning-level sycophancy intensifies sharply in clinical visual judgement, reaching 95.7% for the most affected model. We further introduce a failure taxonomy separating reasoning-chain from answer-level sycophancy, and a complementary sentence-level taxonomy locating where in the chain drift first emerges. Our results show that sycophancy can corrupt the reasoning chain independently of the final answer, so answer-level evaluation alone is insufficient.

Related papers

• Measuring LLM Sycophancy under Sustained Multi-Turn Pressure [16.810854941543926]
  Large language models (LLMs) may abandon correct positions when users push back, exhibiting a failure mode known as sycophancy.<n>We introduce SPINE, a benchmark in which an LLM proxy plays a persistent but mistaken user and adaptively challenges a target model for up to 25 turns.
  arXiv  Detail & Related papers  (2026-09-08T17:35:04Z)
• Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning [56.48520300004217]
  We present Seirnes, a self-play RL framework that transforms contextual interference into an internal training signal.<n>A single model is trained to both construct plausible yet distracting contexts that expose its own reasoning blind spots.<n>By pitting these competing objectives against each other, Seirnes compels the model to move beyond superficial pattern matching.
  arXiv  Detail & Related papers  (2026-05-12T06:58:35Z)
• Ask don't tell: Reducing sycophancy in large language models [1.5701458173528275]
  We show that sycophancy is substantially higher in response to non-questions compared to questions.<n>We find that asking a model to convert non-questions into questions before answering significantly reduces sycophancy.
  arXiv  Detail & Related papers  (2026-02-27T12:27:04Z)
• Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimization [78.94590726578014]
  multimodal reasoning models (MLRMs) remain prone to hallucinations, and effective solutions are still underexplored.<n>We propose C3PO, a training-based mitigation framework comprising textbfCompression and textbfPreference textbfOptimization.
  arXiv  Detail & Related papers  (2026-02-03T11:00:55Z)
• MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models [49.32415342913976]
  We introduce MM-CoT, a diagnostic benchmark designed to probe the visual grounding and logical coherence of CoT reasoning in multimodal models.<n>We evaluate leading vision-language models on MM-CoT and find that even the most advanced systems struggle, revealing a sharp discrepancy between generative fluency and true reasoning fidelity.
  arXiv  Detail & Related papers  (2025-12-09T04:13:31Z)
• MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Models [8.790366364290065]
  Large Reasoning Models (LRMs) suffer from sycophantic behavior, where models tend to agree with users' incorrect beliefs and follow misinformation rather than maintain independent reasoning.<n>Mitigating LRM sycophancy requires monitoring how this sycophancy emerges during the reasoning trajectory.<n>We propose MONICA, a novel Monitor-guided framework that monitors and mitigates sycophancy during model inference at the level of reasoning steps.
  arXiv  Detail & Related papers  (2025-11-09T15:18:58Z)
• Reasoning Models Are More Easily Gaslighted Than You Think [85.84943447589511]
  We evaluate three state-of-the-art reasoning models, including OpenAI's o4-mini, Claude-3.7-Sonnet and Gemini-2.5-Flash.<n>Our evaluation reveals significant accuracy drops following gaslighting negation prompts.<n>We introduce GaslightingBench-R, a new diagnostic benchmark designed to evaluate reasoning models' susceptibility to defend their belief.
  arXiv  Detail & Related papers  (2025-06-11T12:52:25Z)
• What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding [84.42056293290015]
  We analyze the token-level misalignment between reasoning and non-reasoning models.<n>Motivated by the Local Misalignment Diminish, we propose FoReaL-Decoding.<n>On four popular math-reasoning benchmarks, FoReaL-Decoding reduces theoretical FLOPs by 30 to 50% and trims CoT length by up to 40%.
  arXiv  Detail & Related papers  (2025-06-08T05:08:32Z)
• Measuring Sycophancy of Language Models in Multi-turn Dialogues [33.875038658886986]
  We introduce SYCON Bench, a novel benchmark for evaluating sycophancy in multi-turn, free-form conversational settings.<n>Applying SYCON Bench to 17 Large Language Models across three real-world scenarios, we find that sycophancy remains a prevalent failure mode.
  arXiv  Detail & Related papers  (2025-05-28T14:05:46Z)
• Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework [18.54098084470481]
  We analyze sycophancy across vision-language benchmarks and propose an inference-time mitigation framework.<n>Our framework effectively mitigates sycophancy across all evaluated models, while maintaining performance on neutral prompts.
  arXiv  Detail & Related papers  (2024-08-21T01:03:21Z)
• Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures [93.17009514112702]
  Pruning, setting a significant subset of the parameters of a neural network to zero, is one of the most popular methods of model compression. Despite existing evidence for this phenomenon, the relationship between neural network pruning and induced bias is not well-understood.
  arXiv  Detail & Related papers  (2023-04-25T07:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.