Robust image representations with counterfactual contrastive learning

• URL: http://arxiv.org/abs/2409.10365v1
• Date: Mon, 16 Sep 2024 15:11:00 GMT
• Title: Robust image representations with counterfactual contrastive learning
• Authors: Mélanie Roschewitz, Fabio De Sousa Ribeiro, Tian Xia, Galvin Khara, Ben Glocker,
• Abstract summary: We introduce counterfactual contrastive learning, a novel framework leveraging recent advances in causal image synthesis. Our method, evaluated across five datasets, outperforms standard contrastive learning in terms of robustness to acquisition shift. Further experiments show that the proposed framework extends beyond acquisition shifts, with models trained with counterfactual contrastive learning substantially improving subgroup performance across biological sex.
• Score: 17.273155534515393
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive pretraining can substantially increase model generalisation and downstream performance. However, the quality of the learned representations is highly dependent on the data augmentation strategy applied to generate positive pairs. Positive contrastive pairs should preserve semantic meaning while discarding unwanted variations related to the data acquisition domain. Traditional contrastive pipelines attempt to simulate domain shifts through pre-defined generic image transformations. However, these do not always mimic realistic and relevant domain variations for medical imaging such as scanner differences. To tackle this issue, we herein introduce counterfactual contrastive learning, a novel framework leveraging recent advances in causal image synthesis to create contrastive positive pairs that faithfully capture relevant domain variations. Our method, evaluated across five datasets encompassing both chest radiography and mammography data, for two established contrastive objectives (SimCLR and DINO-v2), outperforms standard contrastive learning in terms of robustness to acquisition shift. Notably, counterfactual contrastive learning achieves superior downstream performance on both in-distribution and on external datasets, especially for images acquired with scanners under-represented in the training set. Further experiments show that the proposed framework extends beyond acquisition shifts, with models trained with counterfactual contrastive learning substantially improving subgroup performance across biological sex.

Related papers

• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model [80.61157097223058]
  A prevalent strategy to bolster image classification performance is through augmenting the training set with synthetic images generated by T2I models. In this study, we scrutinize the shortcomings of both current generative and conventional data augmentation techniques. We introduce an innovative inter-class data augmentation method known as Diff-Mix, which enriches the dataset by performing image translations between classes.
  arXiv  Detail & Related papers  (2024-03-28T17:23:45Z)
• Counterfactual contrastive learning: robust representations via causal image synthesis [17.273155534515393]
  CF-SimCLR is a counterfactual contrastive learning approach which leverages approximate counterfactual inference for positive pair creation. We show that CF-SimCLR substantially improves acquisition shift with higher downstream performance on both in- and out-of-distribution data.
  arXiv  Detail & Related papers  (2024-03-14T17:47:01Z)
• Transformer-based Clipped Contrastive Quantization Learning for Unsupervised Image Retrieval [15.982022297570108]
  Unsupervised image retrieval aims to learn the important visual characteristics without any given level to retrieve the similar images for a given query image. In this paper, we propose a TransClippedCLR model by encoding the global context of an image using Transformer having local context through patch based processing. Results using the proposed clipped contrastive learning are greatly improved on all datasets as compared to same backbone network with vanilla contrastive learning.
  arXiv  Detail & Related papers  (2024-01-27T09:39:11Z)
• OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization [65.57380193070574]
  Vision-language pre-training models are vulnerable to multi-modal adversarial examples. Recent works have indicated that leveraging data augmentation and image-text modal interactions can enhance the transferability of adversarial examples. We propose an Optimal Transport-based Adversarial Attack, dubbed OT-Attack.
  arXiv  Detail & Related papers  (2023-12-07T16:16:50Z)
• Unsupervised Domain Transfer with Conditional Invertible Neural Networks [83.90291882730925]
  We propose a domain transfer approach based on conditional invertible neural networks (cINNs) Our method inherently guarantees cycle consistency through its invertible architecture, and network training can efficiently be conducted with maximum likelihood. Our method enables the generation of realistic spectral data and outperforms the state of the art on two downstream classification tasks.
  arXiv  Detail & Related papers  (2023-03-17T18:00:27Z)
• Bridging Synthetic and Real Images: a Transferable and Multiple Consistency aided Fundus Image Enhancement Framework [61.74188977009786]
  We propose an end-to-end optimized teacher-student framework to simultaneously conduct image enhancement and domain adaptation. We also propose a novel multi-stage multi-attention guided enhancement network (MAGE-Net) as the backbones of our teacher and student network.
  arXiv  Detail & Related papers  (2023-02-23T06:16:15Z)
• Metadata-enhanced contrastive learning from retinal optical coherence tomography images [7.932410831191909]
  We extend conventional contrastive frameworks with a novel metadata-enhanced strategy. Our approach employs widely available patient metadata to approximate the true set of inter-image contrastive relationships. Our approach outperforms both standard contrastive methods and a retinal image foundation model in five out of six image-level downstream tasks.
  arXiv  Detail & Related papers  (2022-08-04T08:53:15Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)
• Contrast Adaptive Tissue Classification by Alternating Segmentation and Synthesis [0.21111026813272174]
  We describe an approach using alternating segmentation and synthesis steps that adapts the contrast properties of the training data to the input image. A notable advantage of this approach is that only a single example of the acquisition protocol is required to adapt to its contrast properties.
  arXiv  Detail & Related papers  (2021-03-04T00:25:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.