Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models

• URL: http://arxiv.org/abs/2604.17873v1
• Date: Mon, 20 Apr 2026 06:35:26 GMT
• Title: Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models
• Abstract summary: We identify video syphancy, a failure mode in which Vid-LLMs retract correct, visually grounded judgments under negation-based gaslighting.<n>Rather than merely changing their answers, the models often fabricate unsupported temporal or spatial explanations to justify incorrect revisions.<n>We introduce GasVideo-1000, a curated benchmark designed to probe videotemporal sycophancy with clear visual grounding and temporal reasoning requirements.
• Score: 54.76242207842981
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Large Language Models (Vid-LLMs) have demonstrated remarkable performance in video understanding tasks, yet their robustness under conversational interaction remains largely underexplored. In this paper, we identify spatiotemporal sycophancy, a failure mode in which Vid-LLMs retract initially correct, visually grounded judgments and conform to misleading user feedback under negation-based gaslighting. Rather than merely changing their answers, the models often fabricate unsupported temporal or spatial explanations to justify incorrect revisions. To systematically investigate this phenomenon, we propose a negation-based gaslighting evaluation framework and introduce GasVideo-1000, a curated benchmark designed to probe spatiotemporal sycophancy with clear visual grounding and temporal reasoning requirements. We evaluate a broad range of state-of-the-art open-source and proprietary Vid-LLMs across diverse video understanding tasks. Extensive experiments reveal that vulnerability to negation-based gaslighting is pervasive and severe, even among models with strong baseline performance. While prompt-level grounding constraints can partially mitigate this behavior, they do not reliably prevent hallucinated justifications or belief reversal. Our results indicate that current Vid-LLMs lack robust mechanisms for maintaining grounded spatiotemporal beliefs under adversarial conversational feedback.

Related papers

• Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning [27.676141768874732]
  Video Anomaly Detection (VAD) aims to identify anomalous events and localize their temporal intervals.<n>Existing approaches exhibit a "when-what" dissociation: traditional methods localize when anomalies occur but lack semantic understanding.<n>We propose Glance then Scrutinize (GtS), a training-free framework using static and dynamic textual guidance.<n>Experiments show that GtS substantially surpasses training-free baselines, while the agentic model delivers both higher accuracy and faster inference.
  arXiv  Detail & Related papers  (2026-08-07T17:07:49Z)
• SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs [74.89208610190595]
  We present SPOT-E, a test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning.<n> SPOT-E yields consistent gains and improved robustness under visual corruptions.
  arXiv  Detail & Related papers  (2026-06-18T13:56:30Z)
• LATERN: Test-Time Context-Aware Explainable Video Anomaly Detection [7.614138458711612]
  Vision-language models (VLMs) have emerged as a promising paradigm for video anomaly detection (VAD)<n>In this paper, we aim to address a key limitation of such pipelines, which perform segment-level inference independently owing to token constraints and reason without structured temporal context.<n>We propose a context-aware framework named LATERN, which reformulates VAD as a temporal evidence aggregation process.
  arXiv  Detail & Related papers  (2026-05-14T16:48:03Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification [73.02304272829785]
  VideoBench is a hierarchical benchmark for long-video answering that rigorously verifies evidence.<n>It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence.<n>Experiments show that even GeminiPro-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting.<n>Results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-04-02T03:29:43Z)
• INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs [69.11382230669491]
  hallucinations are outputs that contradict either video evidence (faithfulness) or verifiable world knowledge (factuality)<n>textscINFACT evaluates models in four modes: Base (clean), Visual Degradation, Evidence, and Temporal Intervention for order-sensitive items.<n>Experiments on 14 representative Video-LLMs reveal that higher Base-mode accuracy does not reliably translate to higher reliability in the induced modes.
  arXiv  Detail & Related papers  (2026-03-12T03:03:16Z)
• Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models [18.243585941034116]
  Video-Language Models (VidLMs) robustly account for video content, temporal sequence, and motion.<n>We introduce REVEAL, a diagnostic benchmark that probes fundamental weaknesses of contemporary Vids.<n>We find that these models confidently describe reversed scenes as forward, answer questions while neglecting video content, agree with false claims, struggle with basic camera motion, and fail to aggregate temporal scalable information.
  arXiv  Detail & Related papers  (2026-02-11T17:39:14Z)
• Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision-Language Models [3.701776503593477]
  Modern vision-language models (VLMs) excel at many multimodal tasks, yet their grasp of temporal information in video remains weak and, crucially, under-evaluated.<n>We probe this gap with a deceptively simple but revealing challenge: judging the arrow of time (AoT)-whether a short clip is played forward or backward.<n>We introduce AoT-PsyPhyBENCH, a psychophysically validated benchmark that tests whether VLMs can infer temporal direction in natural videos using the same stimuli and behavioral baselines established for humans.
  arXiv  Detail & Related papers  (2025-10-30T08:21:50Z)
• TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility [70.24211591214528]
  Video generative models produce sequences that violate intuitive physical laws, such as objects floating, teleporting, or morphing.<n>Existing Video-Language Models (VLMs) struggle to identify physics violations, exposing fundamental limitations in their temporal and causal reasoning.<n>We introduce TRAVL, a fine-tuning recipe that combines a balanced training dataset with a trajectory-aware attention module to improve motion encoding.<n>We propose ImplausiBench, a benchmark of 300 videos (150 real, 150 generated) that removes linguistic biases and isolates visual-temporal understanding.
  arXiv  Detail & Related papers  (2025-10-08T21:03:46Z)
• Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs [18.07249962240035]
  Video large language models (Video-LLMs) are increasingly integrated into real-world applications that demand grounded multimodal reasoning.<n>Sycophancy, the tendency of these models to align with user input even when it contradicts the visual evidence, undermines their trustworthiness in such contexts.<n>We propose VISE (Video-LLM Sycophancy Benchmarking and Evaluation), the first benchmark designed to evaluate sycophantic behavior in state-of-the-art Video-LLMs.
  arXiv  Detail & Related papers  (2025-06-08T15:00:21Z)
• Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models [77.96693360763925]
  Video SimpleQA is the first comprehensive benchmark tailored for factuality evaluation in video contexts.<n>Our work differs from existing video benchmarks through the following key features: Knowledge required: demanding integration of external knowledge beyond the video's explicit narrative.<n>Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance.
  arXiv  Detail & Related papers  (2025-03-24T17:46:09Z)
• On the Consistency of Video Large Language Models in Temporal Comprehension [57.985769348320616]
  Video large language models (Video-LLMs) can temporally ground language queries and retrieve video moments.<n>We conduct a study on prediction consistency -- a key indicator for robustness and trustworthiness of temporal grounding.
  arXiv  Detail & Related papers  (2024-11-20T00:47:17Z)
• Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework [18.54098084470481]
  We analyze sycophancy across vision-language benchmarks and propose an inference-time mitigation framework.<n>Our framework effectively mitigates sycophancy across all evaluated models, while maintaining performance on neutral prompts.
  arXiv  Detail & Related papers  (2024-08-21T01:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.