Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation

• URL: http://arxiv.org/abs/2512.24271v1
• Date: Tue, 30 Dec 2025 14:53:33 GMT
• Title: Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation
• Authors: Zhe Huang, Hao Wen, Aiming Hao, Bingze Song, Meiqi Wu, Jiahong Wu, Xiangxiang Chu, Sheng Lu, Haoqian Wang,
• Abstract summary: Multimodal Large Language Models (MLLMs) have made remarkable progress in video understanding.<n>They suffer from a critical vulnerability: an over-reliance on language priors, which can lead to visual ungrounded hallucinations.<n>We introduce DualityForge, a novel data synthesis framework that employs controllable, diffusion-based video editing.
• Score: 37.975475053907545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) have made remarkable progress in video understanding. However, they suffer from a critical vulnerability: an over-reliance on language priors, which can lead to visual ungrounded hallucinations, especially when processing counterfactual videos that defy common sense. This limitation, stemming from the intrinsic data imbalance between text and video, is challenging to address due to the substantial cost of collecting and annotating counterfactual data. To address this, we introduce DualityForge, a novel counterfactual data synthesis framework that employs controllable, diffusion-based video editing to transform real-world videos into counterfactual scenarios. By embedding structured contextual information into the video editing and QA generation processes, the framework automatically produces high-quality QA pairs together with original-edited video pairs for contrastive training. Based on this, we build DualityVidQA, a large-scale video dataset designed to reduce MLLM hallucinations. In addition, to fully exploit the contrastive nature of our paired data, we propose Duality-Normalized Advantage Training (DNA-Train), a two-stage SFT-RL training regime where the RL phase applies pair-wise $\ell_1$ advantage normalization, thereby enabling a more stable and efficient policy optimization. Experiments on DualityVidQA-Test demonstrate that our method substantially reduces model hallucinations on counterfactual videos, yielding a relative improvement of 24.0% over the Qwen2.5-VL-7B baseline. Moreover, our approach achieves significant gains across both hallucination and general-purpose benchmarks, indicating strong generalization capability. We will open-source our dataset and code.

Related papers

• From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs) [2.142303406607558]
  Vision Language Models (VLMs) are poised to revolutionize the digital transformation of pharmacyceutical industry.<n>We introduce a domain adapted Video to Video Clip Generation framework that integrates Audio Language Models (ALMs) and Vision Language Models (VLMs) to produce highlight clips.
  arXiv  Detail & Related papers  (2026-01-08T16:02:56Z)
• Long Grounded Thoughts: Distilling Compositional Visual Reasoning Chains at Scale [70.23466957404891]
  We introduce a new reasoning data generation framework spanning diverse skills and levels of complexity with over 1M high-quality synthetic vision-centric questions.<n>We show that finetuning Qwen2.5-VL-7B on our data outperforms all open-data baselines across all evaluated vision-centric benchmarks.
  arXiv  Detail & Related papers  (2025-11-07T20:50:54Z)
• Generative Video Matting [57.186684844156595]
  Video matting has traditionally been limited by the lack of high-quality ground-truth data.<n>Most existing video matting datasets provide only human-annotated imperfect alpha and foreground annotations.<n>We introduce a novel video matting approach that can effectively leverage the rich priors from pre-trained video diffusion models.
  arXiv  Detail & Related papers  (2025-08-11T12:18:55Z)
• FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers [63.788600404496115]
  FullDiT2 is an efficient in-context conditioning framework for general controllability in both video generation and editing tasks.<n>FullDiT2 achieves significant computation reduction and 2-3 times speedup in averaged time cost per diffusion step.
  arXiv  Detail & Related papers  (2025-06-04T17:57:09Z)
• Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency [56.475612147721264]
  We propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals.<n>We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA.<n>Results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs.
  arXiv  Detail & Related papers  (2025-06-02T17:28:26Z)
• VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning [35.64831081829936]
  Reinforcement fine-tuning (RFT) has shown great promise in achieving humanlevel reasoning capabilities.<n>VideoRFT follows the standard two-stage scheme in RFT: supervised fine-tuning (SFT) with chain-of-thought (CoT) annotations, followed by reinforcement learning (RL) to improve generalization.<n>We show that VideoRFT achieves state-of-the-art performance on six video reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-05-18T14:14:35Z)
• ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations [0.40964539027092917]
  Large Language Models (LLMs) have transformed natural language processing (NLP) tasks, but they suffer from hallucination, generating plausible yet factually incorrect content.<n>This issue extends to Video-Language Models (VideoLLMs), where textual descriptions may inaccurately represent visual content, resulting in multi-modal hallucinations.<n>We introduce a two-step protocol: (1) a faithfulness detection strategy that uses a modified Lynx model to assess semantic alignment between generated captions and ground-truth video references, and (2) a hallucination mitigation strategy using Retrieval-Augmented Generation (RAG) with an ad-hoc knowledge base dynamically constructed during inference.
  arXiv  Detail & Related papers  (2025-04-20T00:10:44Z)
• SkyReels-V2: Infinite-length Film Generative Model [35.00453687783287]
  We propose SkyReels-V2, an Infinite-length Film Generative Model, that synergizes Multi-modal Large Language Model (MLLM), Multi-stage Pretraining, Reinforcement Learning, and Diffusion Forcing Framework.<n>We establish progressive-resolution pretraining for the fundamental video generation, followed by a four-stage post-training enhancement.
  arXiv  Detail & Related papers  (2025-04-17T16:37:27Z)
• VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval [8.908777234657046]
  Large-language and vision-language models (LLM/LVLMs) have gained prominence across various domains.<n>Here we propose VideoLights, a novel HD/MR framework addressing these limitations through (i) Convolutional Projection and Feature Refinement modules.<n> Comprehensive experiments on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-02T14:45:53Z)
• VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection [61.54044967253421]
  We introduce VideoEspresso, a novel dataset that features VideoQA pairs preserving essential spatial details and temporal coherence. Our construction pipeline employs a semantic-aware method to reduce redundancy, followed by generating QA pairs using GPT-4o. We propose a Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a two-stage instruction fine-tuned reasoning LVLM.
  arXiv  Detail & Related papers  (2024-11-22T08:33:36Z)
• EventHallusion: Diagnosing Event Hallucinations in Video LLMs [42.66453293963568]
  Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field.<n>We propose EventHallusion, a novel benchmark that focuses on assessing the VideoLLMs' hallucination toward event.<n>We also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs.
  arXiv  Detail & Related papers  (2024-09-25T03:49:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.