Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

• URL: http://arxiv.org/abs/2607.05850v1
• Date: Tue, 07 Jul 2026 05:13:26 GMT
• Title: Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning
• Abstract summary: Deep neural networks trained with Empirical Risk Minimization often fail under distribution shifts.<n>Recent generative approaches address this issue by creating counterfactual images with altered contexts.<n>We propose a two-stage framework that uses generative intervention to explicitly learn background-invariant visual representations.
• Score: 0.871725390496537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks trained with Empirical Risk Minimization (ERM) often fail under distribution shifts because they exploit spurious correlations between object labels and background context. Recent generative approaches address this issue by creating counterfactual images with altered contexts, but typically use these samples as standard data augmentation, leaving the model free to retain background-sensitive representations. We propose a two-stage framework that uses generative intervention to explicitly learn background-invariant visual representations. First, we isolate the foreground object using zero-shot segmentation and generate context-shifted variants with a structure-preserving diffusion model, preserving object identity while varying the surrounding environment. We then introduce Cross-Variant Self-Supervised Learning, where variants of the same object under different backgrounds form positive pairs in a contrastive objective. This encourages the encoder to align object-centric representations while suppressing background-specific cues. Then, we fine-tune the pretrained encoder using an ERM warm-up followed by GroupDRO with layer-wise learning rates. Experiments on distribution-shift benchmarks demonstrate best worst-group performance, achieving 92.5% on Waterbirds, 81.7% on MetaShift, and 87.4% on NICO++. Code: https://github.com/surajyadav-research/GRSSL

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• Making Training-Free Diffusion Segmentors Scale with the Generative Power [118.72472901404814]
  A line of research focuses on adapting a pre-trained diffusion model to semantic segmentation without any further training.<n>We propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability.
  arXiv  Detail & Related papers  (2026-03-06T11:35:37Z)
• Hierarchical Identity Learning for Unsupervised Visible-Infrared Person Re-Identification [81.3063589622217]
  Unsupervised visible-infrared person re-identification (USVI-ReID) aims to learn modality-invariant image features from unlabeled cross-modal person datasets.
  arXiv  Detail & Related papers  (2025-09-15T05:10:43Z)
• DDB: Diffusion Driven Balancing to Address Spurious Correlations [24.940576844328408]
  Deep neural networks trained with Empirical Risk Minimization often fail to generalize to out-of-distribution samples.<n>We propose a Diffusion Driven Balancing (DDB) technique to generate training samples with text-to-image diffusion models.<n>Our experiments show that our technique achieves better worst-group accuracy than the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-21T15:28:22Z)
• Multilevel Saliency-Guided Self-Supervised Learning for Image Anomaly Detection [15.212031255539022]
  Anomaly detection (AD) is a fundamental task in computer vision. We propose CutSwap, which leverages saliency guidance to incorporate semantic cues for augmentation. CutSwap achieves state-of-the-art AD performance on two mainstream AD benchmark datasets.
  arXiv  Detail & Related papers  (2023-11-30T08:03:53Z)
• SepVAE: a contrastive VAE to separate pathological patterns from healthy ones [2.619659560375341]
  Contrastive Analysis VAE (CA-VAEs) is a family of Variational auto-encoders (VAEs) that aims at separating the common factors of variation between a background dataset (BG) and a target dataset (TG) We show a better performance than previous CA-VAEs methods on three medical applications and a natural images dataset (CelebA)
  arXiv  Detail & Related papers  (2023-07-12T14:52:21Z)
• Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [122.62311703151215]
  Divide and Contrast (DaC) aims to connect the good ends of both worlds while bypassing their limitations. DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals. We further align the source-like domain with the target-specific samples using a memory bank-based Maximum Mean Discrepancy (MMD) loss to reduce the distribution mismatch.
  arXiv  Detail & Related papers  (2022-11-12T09:21:49Z)
• Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing [71.19528222206088]
  We propose a novel Decoupled Multi-task Learning with Cyclical Self-Regulation for face parsing. Specifically, DML-CSR designs a multi-task model which comprises face parsing, binary edge, and category edge detection. Our method achieves the new state-of-the-art performance on the Helen, CelebA-HQ, and LapaMask datasets.
  arXiv  Detail & Related papers  (2022-03-28T02:12:30Z)
• Self-Conditioned Generative Adversarial Networks for Image Editing [61.50205580051405]
  Generative Adversarial Networks (GANs) are susceptible to bias, learned from either the unbalanced data, or through mode collapse. We argue that this bias is responsible not only for fairness concerns, but that it plays a key role in the collapse of latent-traversal editing methods when deviating away from the distribution's core.
  arXiv  Detail & Related papers  (2022-02-08T18:08:24Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)
• Variational Clustering: Leveraging Variational Autoencoders for Image Clustering [8.465172258675763]
  Variational Autoencoders (VAEs) naturally lend themselves to learning data distributions in a latent space. We propose a method based on VAEs where we use a Gaussian Mixture prior to help cluster the images accurately. Our method simultaneously learns a prior that captures the latent distribution of the images and a posterior to help discriminate well between data points.
  arXiv  Detail & Related papers  (2020-05-10T09:34:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.