INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs

• URL: http://arxiv.org/abs/2603.11481v1
• Date: Thu, 12 Mar 2026 03:03:16 GMT
• Title: INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs
• Abstract summary: hallucinations are outputs that contradict either video evidence (faithfulness) or verifiable world knowledge (factuality)<n>textscINFACT evaluates models in four modes: Base (clean), Visual Degradation, Evidence, and Temporal Intervention for order-sensitive items.<n>Experiments on 14 representative Video-LLMs reveal that higher Base-mode accuracy does not reliably translate to higher reliability in the induced modes.
• Score: 69.11382230669491
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite rapid progress, Video Large Language Models (Video-LLMs) remain unreliable due to hallucinations, which are outputs that contradict either video evidence (faithfulness) or verifiable world knowledge (factuality). Existing benchmarks provide limited coverage of factuality hallucinations and predominantly evaluate models only in clean settings. We introduce \textsc{INFACT}, a diagnostic benchmark comprising 9{,}800 QA instances with fine-grained taxonomies for faithfulness and factuality, spanning real and synthetic videos. \textsc{INFACT} evaluates models in four modes: Base (clean), Visual Degradation, Evidence Corruption, and Temporal Intervention for order-sensitive items. Reliability under induced modes is quantified using Resist Rate (RR) and Temporal Sensitivity Score (TSS). Experiments on 14 representative Video-LLMs reveal that higher Base-mode accuracy does not reliably translate to higher reliability in the induced modes, with evidence corruption reducing stability and temporal intervention yielding the largest degradation. Notably, many open-source baselines exhibit near-zero TSS on factuality, indicating pronounced temporal inertia on order-sensitive questions.

Related papers

• Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models [54.76242207842981]
  We identify video syphancy, a failure mode in which Vid-LLMs retract correct, visually grounded judgments under negation-based gaslighting.<n>Rather than merely changing their answers, the models often fabricate unsupported temporal or spatial explanations to justify incorrect revisions.<n>We introduce GasVideo-1000, a curated benchmark designed to probe videotemporal sycophancy with clear visual grounding and temporal reasoning requirements.
  arXiv  Detail & Related papers  (2026-04-20T06:35:26Z)
• Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models [22.554751069064906]
  Recent Video Large Language Models (Video-LLMs) have demonstrated strong capability in video understanding, yet they still suffer from hallucinations.<n>We investigate a decoder-side phenomenon in which the model exhibits a temporally imbalanced concentration pattern.<n>We propose Decoder-side Temporal Rebalancing (DTR), a training-free, layer-selective inference method that rebalances temporal evidence allocation in middle-to-late decoder layers.
  arXiv  Detail & Related papers  (2026-04-14T11:05:42Z)
• LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection [11.420370070522205]
  We introduce LiveFact, a benchmark that simulates the real-world "fog of war" in misinformation detection.<n>LiveFact uses dynamic, temporal evidence sets to evaluate models on their ability to reason with evolving, incomplete information.<n>Tests show that open-source Mixture-of-Experts models, such as Qwen3-235B-A22B, now match or outperform proprietary state-of-the-art systems.
  arXiv  Detail & Related papers  (2026-04-06T16:20:47Z)
• DIQ-H: Evaluating Hallucination Persistence in VLMs Under Temporal Visual Degradation [0.7874708385247353]
  We introduce DIQ-H, the first benchmark for evaluating VLM robustness under dynamic visual degradation in temporal sequences.<n>DIQ-H applies physics-based corruptions including motion blur, sensor noise, and compression artifacts, and measures hallucination persistence, error recovery, and temporal consistency.<n>To enable scalable annotation, we propose Uncertainty-Guided Iterative Refinement (UIR), which generates reliable pseudo-ground-truth.
  arXiv  Detail & Related papers  (2025-12-03T17:22:29Z)
• SVeritas: Benchmark for Robust Speaker Verification under Diverse Conditions [54.34001921326444]
  Speaker verification (SV) models are increasingly integrated into security, personalization, and access control systems.<n>Existing benchmarks evaluate only subsets of these conditions, missing others entirely.<n>We introduce SVeritas, a comprehensive Speaker Verification tasks benchmark suite, assessing SV systems under stressors like recording duration, spontaneity, content, noise, microphone distance, reverberation, channel mismatches, audio bandwidth, codecs, speaker age, and susceptibility to spoofing and adversarial attacks.
  arXiv  Detail & Related papers  (2025-09-21T14:11:16Z)
• SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs [52.03164192840023]
  Large Vision-Language Models (LVLMs) still suffer from hallucinations, i.e., generating content inconsistent with input or established world knowledge.<n>We propose an automated data construction pipeline that produces scalable, controllable, and diverse evaluation data.<n>We construct SHALE, a benchmark designed to assess both faithfulness and factuality hallucinations.
  arXiv  Detail & Related papers  (2025-08-13T07:58:01Z)
• Counterfactual Probing for Hallucination Detection and Mitigation in Large Language Models [0.0]
  We propose Counterfactual Probing, a novel approach for detecting and mitigating hallucinations in large language models.<n>Our method dynamically generates counterfactual statements that appear plausible but contain subtle factual errors, then evaluates the model's sensitivity to these perturbations.
  arXiv  Detail & Related papers  (2025-08-03T17:29:48Z)
• ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs [50.18087419133284]
  hallucination detection methods leveraging hidden states predominantly focus on static and isolated representations.<n>We introduce a novel metric, the ICR Score, which quantifies the contribution of modules to the hidden states' update.<n>We propose a hallucination detection method, the ICR Probe, which captures the cross-layer evolution of hidden states.
  arXiv  Detail & Related papers  (2025-07-22T11:44:26Z)
• Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models [24.363156120809546]
  We propose KIE-HVQA, the first benchmark dedicated to evaluating OCR hallucination in degraded document understanding.<n>This dataset includes test samples spanning identity cards and invoices, with simulated real-world degradations for OCR reliability.<n>Experiments on Qwen2.5-VL demonstrate that our 7B- parameter model achieves a 22% absolute improvement in hallucination-free accuracy over GPT-4o.
  arXiv  Detail & Related papers  (2025-06-25T06:44:07Z)
• VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models [59.05674402770661]
  This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs) VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis.
  arXiv  Detail & Related papers  (2024-06-24T06:21:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.