Towards Deconfounded Image-Text Matching with Causal Inference

• URL: http://arxiv.org/abs/2408.12292v1
• Date: Thu, 22 Aug 2024 11:04:28 GMT
• Title: Towards Deconfounded Image-Text Matching with Causal Inference
• Authors: Wenhui Li, Xinqi Su, Dan Song, Lanjun Wang, Kun Zhang, An-An Liu,
• Abstract summary: We propose an innovative Deconfounded Causal Inference Network (DCIN) for image-text matching task. DCIN decomposes the intra- and inter-modal confounders and incorporates them into the encoding stage of visual and textual features. It can learn causality instead of spurious correlations caused by dataset bias.
• Score: 36.739004282369656
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prior image-text matching methods have shown remarkable performance on many benchmark datasets, but most of them overlook the bias in the dataset, which exists in intra-modal and inter-modal, and tend to learn the spurious correlations that extremely degrade the generalization ability of the model. Furthermore, these methods often incorporate biased external knowledge from large-scale datasets as prior knowledge into image-text matching model, which is inevitable to force model further learn biased associations. To address above limitations, this paper firstly utilizes Structural Causal Models (SCMs) to illustrate how intra- and inter-modal confounders damage the image-text matching. Then, we employ backdoor adjustment to propose an innovative Deconfounded Causal Inference Network (DCIN) for image-text matching task. DCIN (1) decomposes the intra- and inter-modal confounders and incorporates them into the encoding stage of visual and textual features, effectively eliminating the spurious correlations during image-text matching, and (2) uses causal inference to mitigate biases of external knowledge. Consequently, the model can learn causality instead of spurious correlations caused by dataset bias. Extensive experiments on two well-known benchmark datasets, i.e., Flickr30K and MSCOCO, demonstrate the superiority of our proposed method.

Related papers

• Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models [65.82564074712836]
  We introduce DIFfusionHOI, a new HOI detector shedding light on text-to-image diffusion models. We first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions.
  arXiv  Detail & Related papers  (2024-10-26T12:00:33Z)
• Bridging the Modality Gap: Dimension Information Alignment and Sparse Spatial Constraint for Image-Text Matching [10.709744162565274]
  We propose a novel method called DIAS to bridge the modality gap from two aspects. The method achieves 4.3%-10.2% rSum improvements on Flickr30k and MSCOCO benchmarks.
  arXiv  Detail & Related papers  (2024-10-22T09:37:29Z)
• Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective [13.486497323758226]
  Vision-language models pre-trained on extensive datasets can inadvertently learn biases by correlating gender information with objects or scenarios. We propose a framework that incorporates causal mediation analysis to measure and map the pathways of bias generation and propagation.
  arXiv  Detail & Related papers  (2024-07-03T05:19:45Z)
• Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers [120.49126407479717]
  This paper explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos.
  arXiv  Detail & Related papers  (2024-03-12T00:02:03Z)
• Symmetrical Bidirectional Knowledge Alignment for Zero-Shot Sketch-Based Image Retrieval [69.46139774646308]
  This paper studies the problem of zero-shot sketch-based image retrieval (ZS-SBIR) It aims to use sketches from unseen categories as queries to match the images of the same category. We propose a novel Symmetrical Bidirectional Knowledge Alignment for zero-shot sketch-based image retrieval (SBKA)
  arXiv  Detail & Related papers  (2023-12-16T04:50:34Z)
• Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation Learning and Retrieval [12.30468719055037]
  A Contrastive Cross-Modal Knowledge Sharing Pre-training (COOKIE) is developed to grasp the joint text-image representations. The first module is a weight-sharing transformer that builds on the head of the visual and textual encoders. The other one is three specially designed contrastive learning, aiming to share knowledge between different models.
  arXiv  Detail & Related papers  (2022-07-02T04:08:44Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Uncovering Main Causalities for Long-tailed Information Extraction [14.39860866665021]
  Long-tailed distributions caused by the selection bias of a dataset may lead to incorrect correlations. This motivates us to propose counterfactual IE (CFIE), a novel framework that aims to uncover the main causalities behind data.
  arXiv  Detail & Related papers  (2021-09-11T08:08:24Z)
• Generative Interventions for Causal Learning [27.371436971655303]
  We introduce a framework for learning robust visual representations that generalize to new viewpoints, backgrounds, and scene contexts. We show that we can steer generative models to manufacture interventions on features caused by confounding factors.
  arXiv  Detail & Related papers  (2020-12-22T16:01:55Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.