MACD: Model-Aware Contrastive Decoding via Counterfactual Data

• URL: http://arxiv.org/abs/2602.01740v2
• Date: Fri, 06 Feb 2026 19:12:11 GMT
• Title: MACD: Model-Aware Contrastive Decoding via Counterfactual Data
• Authors: Qixin Xiao,
• Abstract summary: Video language models (Video-LLMs) are prone to hallucinations, often generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased.<n>We propose Model-aware Counterfactual Data based Contrastive Decoding (MACD), a new inference strategy that combines model-guided counterfactual construction with decoding.<n>Our approach uses the Video-LLM's own feedback to identify object regions most responsible for hallucination, generating targeted counterfactual inputs at the object level rather than arbitrary frame or temporal modifications.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video language models (Video-LLMs) are prone to hallucinations, often generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased. Existing decoding methods, such as contrastive decoding (CD), rely on random perturbations to construct contrastive data for mitigating hallucination patterns. However, such a way is hard to control the visual cues that drive hallucination or well align with model weaknesses. We propose Model-aware Counterfactual Data based Contrastive Decoding (MACD), a new inference strategy that combines model-guided counterfactual construction with decoding. Our approach uses the Video-LLM's own feedback to identify object regions most responsible for hallucination, generating targeted counterfactual inputs at the object level rather than arbitrary frame or temporal modifications. These model-aware counterfactual data is then integrated into CD to enforce evidence-grounded token selection during decoding. Experiments on EventHallusion, MVBench, Perception-test and Video-MME show that MACD consistently reduces hallucination while maintaining or improving task accuracy across diverse Video-LLMs, including Qwen and InternVL families. The method is especially effective in challenging scenarios involving small, occluded, or co-occurring objects. Our code and data will be publicly released.

Related papers

• Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models [44.84227796501077]
  We introduce OmniVCHall, a benchmark designed to evaluate both isolated and compositional hallucinations in video multimodal large language models (VLLMs)<n>We propose TriCD, a contrastive decoding framework with a triple-pathway calibration mechanism.<n> Experimental results show that TriCD consistently improves performance across two representative backbones, achieving an average accuracy improvement of over 10%.
  arXiv  Detail & Related papers  (2026-01-31T06:50:43Z)
• SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination [48.601385640941935]
  We propose SAVE, a framework that mitigates hallucination by steering the model along Sparse Autoencoder latent features.<n>A binary object-presence question-answering probe identifies the SAE features most indicative of the model's visual information processing.<n>With its simple design, SAVE outperforms state-of-the-art training-free methods on standard benchmarks.
  arXiv  Detail & Related papers  (2025-12-08T17:20:07Z)
• SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding [30.820850789099932]
  We propose a training-free method that adaptively enhances temporal and spatial faithfulness for each output token.<n>SEASON outperforms all existing training-free hallucination mitigation approaches on three hallucination examination benchmarks.
  arXiv  Detail & Related papers  (2025-12-04T10:17:20Z)
• MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding [53.068815533016355]
  We propose image head Masked Contrastive Decoding (MaskCD) for large vision-language models (LVLMs)<n>Our approach utilizes the "image heads" in LVLMs, masking them to construct contrastive samples for contrastive decoding.<n>The results demonstrate that MaskCD effectively alleviates the phenomenon of hallucinations and retains the general capabilities of LVLMs.
  arXiv  Detail & Related papers  (2025-10-03T07:59:16Z)
• MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models [56.49314029765706]
  We introduce MESH, a benchmark designed to evaluate hallucinations in LVMs systematically.<n>MESH uses a Question-Answering framework with binary and multi-choice formats incorporating target and trap instances.<n>We demonstrate that MESH offers an effective and comprehensive approach for identifying hallucinations in videos.
  arXiv  Detail & Related papers  (2025-09-10T12:34:07Z)
• AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding [38.71842806548495]
  We propose Audio-Visual Contrastive Decoding (AVCD) to model trimodal interactions and suppress hallucinations in large language models (MLLMs)<n>To improve efficiency, we introduce entropy-guided adaptive decoding, which skips unnecessary decoding steps based on the model's confidence in its predictions.
  arXiv  Detail & Related papers  (2025-05-27T08:13:57Z)
• Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models [39.9447198156097]
  Mixture of Decoding (MoD) is a novel approach for hallucination mitigation.<n>It adapts decoding strategies by evaluating the correctness of the model's attention on image tokens.<n>MoD significantly outperforms existing decoding methods across multiple mainstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-17T09:44:18Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• EventHallusion: Diagnosing Event Hallucinations in Video LLMs [42.66453293963568]
  Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field.<n>We propose EventHallusion, a novel benchmark that focuses on assessing the VideoLLMs' hallucination toward event.<n>We also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs.
  arXiv  Detail & Related papers  (2024-09-25T03:49:46Z)
• VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs [64.60035916955837]
  VANE-Bench is a benchmark designed to assess the proficiency of Video-LMMs in detecting anomalies and inconsistencies in videos.<n>Our dataset comprises an array of videos synthetically generated using existing state-of-the-art text-to-video generation models.<n>We evaluate nine existing Video-LMMs, both open and closed sources, on this benchmarking task and find that most of the models encounter difficulties in effectively identifying the subtle anomalies.
  arXiv  Detail & Related papers  (2024-06-14T17:59:01Z)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding [125.05295513481035]
  We introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families.
  arXiv  Detail & Related papers  (2023-11-28T16:26:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.