GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them

• URL: http://arxiv.org/abs/2609.06058v1
• Date: Sat, 05 Sep 2026 12:30:39 GMT
• Title: GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them
• Abstract summary: We introduce GradeTrap, a controlled evaluation that places two social cues in direct conflict.<n>A student answer should attract sycophantic agreement, and a conflicting answer attributed to a peer, teacher, or official answer key should attract authority-based deference.<n>We test the models on 60 synthetic real-world trade-off scenarios.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As vision-language models (VLMs) become increasingly capable and are deployed in consequential real-world settings, they must evaluate evidence independently rather than defer uncritically to human authority. We introduce GradeTrap, a controlled evaluation that places two social cues in direct conflict: a student answer, which should attract sycophantic agreement, and a conflicting answer attributed to a peer, teacher, or official answer key, which should attract authority-based deference. Models produce free-form answers while being explicitly instructed to solve independently and ignore all student answers, feedback, and grading marks. We test the models on 60 synthetic real-world trade-off scenarios. Five neutral trials establish a stable model-relative preference, followed by three repetitions of six experimental cues including controls. On the 45-item common intersection across Gemini 3.5 Flash-Lite, GPT-5.6 Luna, and Claude Haiku 4.5, a generic second-answer control yields 5.4% conflicting-answer selection. Relative to that control, pooled within-item changes show no reliable peer-review effect, a 6.9-point teacher-review effect, and a 19.5-point official-key effect. In contrast, a displayed conflicting student answer alone compared to a displayed student reference answer alone only raises selection from 2.2% to 5.2%. Official-key provenance therefore redirects judgements more than a student answer or the generic second-answer control, despite an explicit ignore instruction and an opposing student answer given along with the official key. Effects vary in magnitude across the three models.

Related papers

• Self-Supervised Visual On-Policy Distillation [63.83705343579541]
  Visual on-policy distillation relies heavily on an informative teacher-student asymmetry.<n>We introduce Self-Supervised Visual On-Policy Distillation (S$2$VOPD)<n>S$2$VOPD distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image.
  arXiv  Detail & Related papers  (2026-08-14T09:52:01Z)
• Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior [0.0]
  An effective large language model (LLM) tutor must often decline to give an answer it could easily produce.<n>We report a deployed tutoring system that enforces answer-withholding as a per-turn, machine-checkable contract.<n>We tune the behavior with an automated evaluation that uses no human subjects.
  arXiv  Detail & Related papers  (2026-08-12T17:35:58Z)
• Divergent Response Modes in Frontier Language Models Under Steering Pressure [0.0]
  This study evaluates behavioral steerability across six frontier models from six developers using 300 paired base and steered items over three categories: values-conflict, reasoning-elicitation, and reasoning-suppression.<n>We find that models differ not just in how much steering shifts their behavior but in what kind of response they give, and some response modes appear in only one or two of them.
  arXiv  Detail & Related papers  (2026-08-06T20:48:14Z)
• Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models [15.860351997992042]
  LLM tutoring poses a measurement problem: can a general-purpose helpfulness rubric distinguish direct answer-giving from pedagogical guidance?<n>We compare conversational and pedagogical policies instantiated with the same underlying model and paired with one weak simulated student.
  arXiv  Detail & Related papers  (2026-07-30T12:37:28Z)
• Visual Contrastive Self-Distillation [61.298159957622886]
  We propose Visual Contrastive Self-Distillation, which converts image-content removal into an on-policy self-distillation signal.<n>We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student.
  arXiv  Detail & Related papers  (2026-07-23T17:37:37Z)
• OPOD: On-Policy Omni Distillation [55.440058537827134]
  Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
  arXiv  Detail & Related papers  (2026-07-23T04:55:04Z)
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals [65.71225905458182]
  Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher.<n>This setup provides dense token-level feedback without relying on a separate external model.<n>We introduce AVSD, a novel method of self-distillation with multiple privileged-information views.
  arXiv  Detail & Related papers  (2026-05-20T03:06:36Z)
• Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators [55.617099475539305]
  Large language models (LLMs) can fluently generate student-like responses, making them attractive as simulated students for training and evaluating AI tutors and human educators.<n>Yet such simulators are typically evaluated by output similarity to real students, not by whether they behave like students with coherent misconceptions during interaction.<n>We introduce a controlled framework for evaluating misconception faithfulness, whether a simulator maintains a misconception-driven belief state and updates selectively when feedback addresses the underlying misconception.
  arXiv  Detail & Related papers  (2026-05-12T20:55:23Z)
• The Fragility Of Moral Judgment In Large Language Models [0.0]
  We introduce a perturbation framework for testing the stability and manipulability of large language models (LLMs)<n>We generate three families of content perturbations: surface edits (lexical/structural noise), point-of-view shifts (voice and stance neutralization), and persuasion cues.<n>Surface perturbations produce low flip rates (7.5%), largely within the self-consistency noise floor.<n>Instability concentrates in morally ambiguous cases; scenarios where no party is assigned blame are most susceptible.
  arXiv  Detail & Related papers  (2026-03-05T20:01:43Z)
• The Judge Who Never Admits: Hidden Shortcuts in LLM-based Evaluation [17.386684382460242]
  Large language models (LLMs) are increasingly used to evaluate system outputs in tasks such as reasoning, question answering, and creative writing.<n>We test this ideal via controlled cue perturbations-synthetic metadata labels injected into evaluation prompts for six judge models.<n>We study six cue families: source, temporal, age, gender, ethnicity, and educational status.
  arXiv  Detail & Related papers  (2026-02-08T14:45:23Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.