Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decomposition

• URL: http://arxiv.org/abs/2604.05279v1
• Date: Tue, 07 Apr 2026 00:28:17 GMT
• Title: Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decomposition
• Abstract summary: We operationalise sycophancy through formal definitions of pressure independence and evidence responsiveness.<n>We propose the first approach to sycophancy reduction via reward decomposition.
• Score: 4.5328321409938095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models exhibit sycophancy, the tendency to shift their stated positions toward perceived user preferences or authority cues regardless of evidence. Standard alignment methods fail to correct this because scalar reward models conflate two distinct failure modes into a single signal: pressure capitulation, where the model changes a correct answer under social pressure, and evidence blindness, where the model ignores the provided context entirely. We operationalise sycophancy through formal definitions of pressure independence and evidence responsiveness, serving as a working framework for disentangled training rather than a definitive characterisation of the phenomenon. We propose the first approach to sycophancy reduction via reward decomposition, introducing a multi-component Group Relative Policy Optimisation (GRPO) reward that decomposes the training signal into five terms: pressure resistance, context fidelity, position consistency, agreement suppression, and factual correctness. We train using a contrastive dataset pairing pressure-free baselines with pressured variants across three authority levels and two opposing evidence contexts. Across five base models, our two-phase pipeline consistently reduces sycophancy on all metric axes, with ablations confirming that each reward term governs an independent behavioural dimension. The learned resistance to pressure generalises beyond our training methodology and prompt structure, reducing answer-priming sycophancy by up to 17 points on SycophancyEval despite the absence of such pressure forms during training.

Related papers

• Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models [62.522141464687195]
  We introduce Action-JND, which extends the principle of just noticeable difference (JND) to embodied perception.<n>A token change is considered tolerable only when the induced action deviation remains within a tolerated margin.<n>The resulting action-tolerance score serves as a plug-and-play criterion for VLA compression paradigms.
  arXiv  Detail & Related papers  (2026-08-21T15:59:37Z)
• Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure [6.892220883660279]
  We introduce a benchmark and dataset for evaluating LMRM sycophancy when confronted with a wrong answer from a user.<n>We find that sycophancy is prevalent under pressure, with Statement eliciting the highest rates and Conviction the lowest.<n>We introduce a failure taxonomy separating reasoning-chain from answer-level sycophancy, and a complementary sentence-level taxonomy locating where in the chain drift first emerges.
  arXiv  Detail & Related papers  (2026-07-30T03:45:46Z)
• RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation [46.554172283247915]
  On-policy self-distillation provides dense, token-level supervision for reasoning models.<n>We show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones.<n>We propose textbfRLCSD (Reinforcement Learning with Contrastive on-policy Self-Distillation) to mitigate this drift.
  arXiv  Detail & Related papers  (2026-06-10T06:31:59Z)
• Beyond Social Pressure: Benchmarking Epistemic Attack in Large Language Models [0.07812854697536452]
  We develop a diagnostic benchmark for evaluating textitepistemic attack<n> prompts challenge the legitimacy of knowledge, values, or identity rather than simply opposing a previous answer.
  arXiv  Detail & Related papers  (2026-04-09T03:14:30Z)
• SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy [1.4732811715354452]
  SWAY is an unsupervised computational linguistic measure of sycophancy.<n>We develop a counterfactual prompting mechanism to identify how much a model's agreement shifts under positive or negative linguistic pressure.<n>We introduce a counterfactual mitigation strategy teaching models to consider what the answer would be if opposite assumptions were suggested.
  arXiv  Detail & Related papers  (2026-04-02T18:00:14Z)
• Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Models [6.431677598656395]
  In contested domains, instruction-tuned language models must balance user-alignment pressures against faithfulness to the in-context evidence.<n>We conduct fine-grained ablations over evidence composition and uncertainty cues across 19 instruction-tuned models spanning 0.27B to 32B parameters.
  arXiv  Detail & Related papers  (2026-03-20T17:38:23Z)
• Why Agents Compromise Safety Under Pressure [6.429973383484847]
  This paper identifies a new concept called Agentic Pressure, which characterizes the endogenous tension emerging when compliant execution becomes infeasible.<n>We demonstrate that under this pressure agents exhibit normative drift where they strategically sacrifice safety to preserve utility.<n>We find that advanced reasoning capabilities accelerate this decline as models construct linguistic rationalizations to justify violation.
  arXiv  Detail & Related papers  (2026-03-16T08:37:34Z)
• Beacon: Single-Turn Diagnosis and Mitigation of Latent Sycophancy in Large Language Models [0.0]
  Large language models internalize a structural trade-off between truthfulness and obsequious flattery.<n>This latent bias, known as sycophancy, manifests as a preference for user agreement over principled reasoning.<n>We introduce Beacon, a single-turn forced-choice benchmark that isolates this bias independent of conversational context.
  arXiv  Detail & Related papers  (2025-10-19T06:36:57Z)
• Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization [72.30168853571216]
  multimodal large language models excel at tasks that integrate visual perception with symbolic reasoning.<n>CapPO integrates two key mechanisms: (1) a caption-based consistency regularization, which minimizes the divergence between responses conditioned on raw images and those conditioned on captions, and (2) a KL-weighted advantage estimation scheme, which adaptively scales reinforcement signals to strengthen perceptually consistent trajectories.
  arXiv  Detail & Related papers  (2025-09-26T04:32:26Z)
• Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QA [36.21980066799023]
  sycophancy is the tendency to align with user beliefs regardless of correctness.<n>Despite its importance, sycophancy remains underexamined in factual question answering contexts.<n>We introduce a unified evaluation framework to quantify the impact of sycophantic context on model behavior.
  arXiv  Detail & Related papers  (2025-08-19T11:30:52Z)
• Conditioning Matters: Training Diffusion Policies is Faster Than You Think [69.31534053485711]
  Diffusion policies have emerged as a mainstream paradigm for building vision-language-action (VLA) models.<n>We identify a fundamental challenge in conditional diffusion policy training: when generative conditions are hard to distinguish, the training objective degenerates into modeling the marginal action distribution.<n>We propose Cocos, a solution that modifies the source distribution in the conditional flow matching to be condition-dependent.
  arXiv  Detail & Related papers  (2025-05-16T11:14:22Z)
• Counterfactual Realizability [52.85109506684737]
  We introduce a formal definition of realizability, the ability to draw samples from a distribution, and then develop a complete algorithm to determine whether an arbitrary counterfactual distribution is realizable.<n>We illustrate the implications of this new framework for counterfactual data collection using motivating examples from causal fairness and causal reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-14T20:54:27Z)
• Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
  We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
  arXiv  Detail & Related papers  (2023-11-03T03:18:40Z)
• Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts [61.929388479847525]
  This paper focuses on the problem of detecting and reacting to changes in the distribution of a sensorimotor controller's observables. The key idea is the design of switching policies that can take conformal quantiles as input. We show how to design such policies by using conformal quantiles to switch between base policies with different characteristics.
  arXiv  Detail & Related papers  (2023-11-02T17:59:30Z)
• Federated Conformal Predictors for Distributed Uncertainty Quantification [83.50609351513886]
  Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning. In this paper, we extend conformal prediction to the federated learning setting. We propose a weaker notion of partial exchangeability, better suited to the FL setting, and use it to develop the Federated Conformal Prediction framework.
  arXiv  Detail & Related papers  (2023-05-27T19:57:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.