Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention

• URL: http://arxiv.org/abs/2410.10184v1
• Date: Mon, 14 Oct 2024 06:09:16 GMT
• Title: Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention
• Authors: Ying Liu, Ge Bai, Chenji Lu, Shilong Li, Zhang Zhang, Ruifang Liu, Wenbin Guo,
• Abstract summary: We propose a novel causal intervention training scheme named CIBi to eliminate language bias from a finer-grained perspective. We employ causal intervention and contrastive learning to eliminate context bias and improve the multi-modal representation. We design a new question-only branch based on counterfactual generation to distill and eliminate keyword bias.
• Score: 9.859335795616028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the remarkable advancements in Visual Question Answering (VQA), the challenge of mitigating the language bias introduced by textual information remains unresolved. Previous approaches capture language bias from a coarse-grained perspective. However, the finer-grained information within a sentence, such as context and keywords, can result in different biases. Due to the ignorance of fine-grained information, most existing methods fail to sufficiently capture language bias. In this paper, we propose a novel causal intervention training scheme named CIBi to eliminate language bias from a finer-grained perspective. Specifically, we divide the language bias into context bias and keyword bias. We employ causal intervention and contrastive learning to eliminate context bias and improve the multi-modal representation. Additionally, we design a new question-only branch based on counterfactual generation to distill and eliminate keyword bias. Experimental results illustrate that CIBi is applicable to various VQA models, yielding competitive performance.

Related papers

• GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models [75.04426753720553]
  We propose a framework to identify, quantify, and explain biases in an open set setting. This pipeline leverages a Large Language Model (LLM) to propose biases starting from a set of captions. We show two variations of this framework: OpenBias and GradBias.
  arXiv  Detail & Related papers  (2024-08-29T16:51:07Z)
• Mitigating Bias for Question Answering Models by Tracking Bias Influence [84.66462028537475]
  We propose BMBI, an approach to mitigate the bias of multiple-choice QA models. Based on the intuition that a model would lean to be more biased if it learns from a biased example, we measure the bias level of a query instance. We show that our method could be applied to multiple QA formulations across multiple bias categories.
  arXiv  Detail & Related papers  (2023-10-13T00:49:09Z)
• Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA [111.41719652451701]
  We first model a confounding effect that causes language and vision bias simultaneously. We then propose a counterfactual inference to remove the influence of this effect. The proposed method outperforms the state-of-the-art methods in VQA-CP v2 datasets.
  arXiv  Detail & Related papers  (2023-05-31T09:02:58Z)
• Discovering and Mitigating Visual Biases through Keyword Explanation [66.71792624377069]
  We propose the Bias-to-Text (B2T) framework, which interprets visual biases as keywords. B2T can identify known biases, such as gender bias in CelebA, background bias in Waterbirds, and distribution shifts in ImageNet-R/C. B2T uncovers novel biases in larger datasets, such as Dollar Street and ImageNet.
  arXiv  Detail & Related papers  (2023-01-26T13:58:46Z)
• Differential Bias: On the Perceptibility of Stance Imbalance in Argumentation [35.2494622378896]
  We ask whether an "absolute bias classification" is a promising goal at all. To decide whether a text has crossed the proverbial line between non-biased and biased is subjective. A prerequisite for this kind of bias model is the ability of humans to perceive relative bias differences in the first place.
  arXiv  Detail & Related papers  (2022-10-13T12:48:07Z)
• Power of Explanations: Towards automatic debiasing in hate speech detection [19.26084350822197]
  Hate speech detection is a common downstream application of natural language processing (NLP) in the real world. We propose an automatic misuse detector (MiD) relying on an explanation method for detecting potential bias.
  arXiv  Detail & Related papers  (2022-09-07T14:14:03Z)
• Challenges in Measuring Bias via Open-Ended Language Generation [1.5552869983952944]
  We analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We provide recommendations for reporting biases in open-ended language generation.
  arXiv  Detail & Related papers  (2022-05-23T19:57:15Z)
• The SAME score: Improved cosine based bias score for word embeddings [49.75878234192369]
  We introduce SAME, a novel bias score for semantic bias in embeddings. We show that SAME is capable of measuring semantic bias and identify potential causes for social bias in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-28T09:28:13Z)
• Counterfactual VQA: A Cause-Effect Look at Language Bias [117.84189187160005]
  VQA models tend to rely on language bias as a shortcut and fail to sufficiently learn the multi-modal knowledge from both vision and language. We propose a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers.
  arXiv  Detail & Related papers  (2020-06-08T01:49:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.