Deconfounded Image Captioning: A Causal Retrospect

• URL: http://arxiv.org/abs/2003.03923v2
• Date: Fri, 1 Dec 2023 06:24:12 GMT
• Title: Deconfounded Image Captioning: A Causal Retrospect
• Abstract summary: We present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0. DICv1.0 is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split
• Score: 89.0694058049884
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.

Related papers

• SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision [59.61988843996952]
  Style-Aware Visual Early Revision SAVER is a novel mechanism that dynamically adjusts LVLMs' final outputs based on the token-level visual attention patterns.<n>We show that SAVER achieves state-of-the-art performance in hallucination mitigation across various models, datasets, and tasks.
  arXiv  Detail & Related papers  (2025-08-05T07:41:25Z)
• Impact of Language Guidance: A Reproducibility Study [0.0]
  Recent advances in self-supervised learning allow us to train huge models without explicit annotation. We use an off-the-shelf image captioning model, BLIP-2, to replace the captions and improve performance. We also devise a new metric to evaluate the semantic capabilities of self-supervised models based on interpretability methods.
  arXiv  Detail & Related papers  (2025-04-10T21:59:13Z)
• Towards Deconfounded Image-Text Matching with Causal Inference [36.739004282369656]
  We propose an innovative Deconfounded Causal Inference Network (DCIN) for image-text matching task. DCIN decomposes the intra- and inter-modal confounders and incorporates them into the encoding stage of visual and textual features. It can learn causality instead of spurious correlations caused by dataset bias.
  arXiv  Detail & Related papers  (2024-08-22T11:04:28Z)
• Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective [13.486497323758226]
  Vision-language models pre-trained on extensive datasets can inadvertently learn biases by correlating gender information with objects or scenarios. We propose a framework that incorporates causal mediation analysis to measure and map the pathways of bias generation and propagation.
  arXiv  Detail & Related papers  (2024-07-03T05:19:45Z)
• DRStageNet: Deep Learning for Diabetic Retinopathy Staging from Fundus Images [3.4456298317539313]
  Timely identification is critical to curb vision impairment. Models often fail to generalize due to distribution shifts between the source domain on which the model was trained and the target domain where it is deployed. We introduce DRStageNet, a deep learning model designed to mitigate this challenge.
  arXiv  Detail & Related papers  (2023-12-22T18:09:20Z)
• Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning [90.13978453378768]
  We introduce a comprehensive typology of factual errors in generated chart captions. A large-scale human annotation effort provides insight into the error patterns and frequencies in captions crafted by various chart captioning models. Our analysis reveals that even state-of-the-art models, including GPT-4V, frequently produce captions laced with factual inaccuracies.
  arXiv  Detail & Related papers  (2023-12-15T19:16:21Z)
• Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention [72.12974259966592]
  We present a unique and systematic study of a temporal bias due to frame length discrepancy between training and test sets of trimmed video clips. We propose a causal debiasing approach and perform extensive experiments and ablation studies on the Epic-Kitchens-100, YouCook2, and MSR-VTT datasets.
  arXiv  Detail & Related papers  (2023-09-17T15:58:27Z)
• Uncurated Image-Text Datasets: Shedding Light on Demographic Bias [21.421722941901123]
  Even small but manually annotated datasets, such as MSCOCO, are affected by societal bias. Our first contribution is to annotate part of the Google Conceptual Captions dataset, widely used for training vision-and-language models. Second contribution is to conduct a comprehensive analysis of the annotations, focusing on how different demographic groups are represented. Third contribution is to evaluate three prevailing vision-and-language tasks, showing that societal bias is a persistent problem in all of them.
  arXiv  Detail & Related papers  (2023-04-06T02:33:51Z)
• HIVE: Harnessing Human Feedback for Instructional Visual Editing [127.29436858998064]
  We present a novel framework to harness human feedback for instructional visual editing (HIVE) Specifically, we collect human feedback on the edited images and learn a reward function to capture the underlying user preferences. We then introduce scalable diffusion model fine-tuning methods that can incorporate human preferences based on the estimated reward.
  arXiv  Detail & Related papers  (2023-03-16T19:47:41Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)
• Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders [81.30960319178725]
  We propose DivNoising, a denoising approach based on fully convolutional variational autoencoders (VAEs) First we introduce a principled way of formulating the unsupervised denoising problem within the VAE framework by explicitly incorporating imaging noise models into the decoder. We show that such a noise model can either be measured, bootstrapped from noisy data, or co-learned during training.
  arXiv  Detail & Related papers  (2020-06-10T21:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.