Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention

• URL: http://arxiv.org/abs/2511.10268v1
• Date: Fri, 14 Nov 2025 01:42:25 GMT
• Title: Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention
• Authors: Zhe Xu, Zhicai Wang, Junkang Wu, Jinda Lu, Xiang Wang,
• Abstract summary: Large Vision-Language Models (LVLMs) often suffer from object hallucination, making erroneous judgments about the presence of objects in images.<n>We introduce causal analysis into the object recognition scenario of LVLMs, establishing a Structural Causal Model (SCM)<n>We develop Causal-HalBench, a benchmark specifically constructed with counterfactual samples and integrated with comprehensive causal metrics.
• Score: 18.08509160653814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (LVLMs) often suffer from object hallucination, making erroneous judgments about the presence of objects in images. We propose this primar- ily stems from spurious correlations arising when models strongly associate highly co-occurring objects during train- ing, leading to hallucinated objects influenced by visual con- text. Current benchmarks mainly focus on hallucination de- tection but lack a formal characterization and quantitative evaluation of spurious correlations in LVLMs. To address this, we introduce causal analysis into the object recognition scenario of LVLMs, establishing a Structural Causal Model (SCM). Utilizing the language of causality, we formally de- fine spurious correlations arising from co-occurrence bias. To quantify the influence induced by these spurious correla- tions, we develop Causal-HalBench, a benchmark specifically constructed with counterfactual samples and integrated with comprehensive causal metrics designed to assess model ro- bustness against spurious correlations. Concurrently, we pro- pose an extensible pipeline for the construction of these coun- terfactual samples, leveraging the capabilities of proprietary LVLMs and Text-to-Image (T2I) models for their genera- tion. Our evaluations on mainstream LVLMs using Causal- HalBench demonstrate these models exhibit susceptibility to spurious correlations, albeit to varying extents.

Related papers

• CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching [50.65932158912512]
  We propose a new causal reasoning benchmark, CausalFlip, to encourage the development of new large language models.<n>CaulFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations.<n>We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought supervision, and a proposed internalized causal reasoning approach.
  arXiv  Detail & Related papers  (2026-02-23T18:06:15Z)
• Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs [36.89422086121058]
  We show that errors result from a structured yet flawed mechanism that we term class-based (mis)generalization.<n>Experiments on Llama-3, Mistral, and Pythia reveal that this behavior is reflected in the model's internal computations.
  arXiv  Detail & Related papers  (2025-05-28T17:47:52Z)
• Mitigating Spurious Correlations with Causal Logit Perturbation [22.281052412112263]
  This study introduces a novel Causal Logit Perturbation (CLP) framework to train classifiers with generated causal logit perturbations for individual samples.<n>The framework is optimized by an online meta-learning-based learning algorithm and leverages human causal knowledge by augmenting metadata in both counterfactual and factual manners.
  arXiv  Detail & Related papers  (2025-05-21T08:21:02Z)
• A Causal Adjustment Module for Debiasing Scene Graph Generation [28.44150555570101]
  We employ causal inference techniques to model the causality among skewed distributions.<n>Our method enables the composition of zero-shot relationships, thereby enhancing the model's ability to recognize such relationships.
  arXiv  Detail & Related papers  (2025-03-22T20:44:01Z)
• Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors [74.04775677110179]
  In-context Learning (ICL) has become the primary method for performing natural language tasks with Large Language Models (LLMs)<n>In this work, we examine whether this is the result of the aggregation used in corresponding datasets, where trying to combine low-agreement, disparate annotations might lead to annotation artifacts that create detrimental noise in the prompt.<n>Our results indicate that aggregation is a confounding factor in the modeling of subjective tasks, and advocate focusing on modeling individuals instead.
  arXiv  Detail & Related papers  (2024-10-17T17:16:00Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)
• Identifiable Latent Polynomial Causal Models Through the Lens of Change [82.14087963690561]
  Causal representation learning aims to unveil latent high-level causal representations from observed low-level data.<n>One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability.
  arXiv  Detail & Related papers  (2023-10-24T07:46:10Z)
• Context De-confounded Emotion Recognition [12.037240778629346]
  Context-Aware Emotion Recognition (CAER) aims to perceive the emotional states of the target person with contextual information. A long-overlooked issue is that a context bias in existing datasets leads to a significantly unbalanced distribution of emotional states. This paper provides a causality-based perspective to disentangle the models from the impact of such bias, and formulate the causalities among variables in the CAER task.
  arXiv  Detail & Related papers  (2023-03-21T15:12:20Z)
• On Disentangled Representations Learned From Correlated Data [59.41587388303554]
  We bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data. We show that systematically induced correlations in the dataset are being learned and reflected in the latent representations. We also demonstrate how to resolve these latent correlations, either using weak supervision during training or by post-hoc correcting a pre-trained model with a small number of labels.
  arXiv  Detail & Related papers  (2020-06-14T12:47:34Z)
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder [52.139696854386976]
  The framework of variational autoencoder (VAE) is commonly used to disentangle independent factors from observations. We propose a new VAE based framework named CausalVAE, which includes a Causal Layer to transform independent factors into causal endogenous ones. Results show that the causal representations learned by CausalVAE are semantically interpretable, and their causal relationship as a Directed Acyclic Graph (DAG) is identified with good accuracy.
  arXiv  Detail & Related papers  (2020-04-18T20:09:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.