Training Debiased Subnetworks with Contrastive Weight Pruning

• URL: http://arxiv.org/abs/2210.05247v3
• Date: Mon, 26 Jun 2023 10:40:08 GMT
• Title: Training Debiased Subnetworks with Contrastive Weight Pruning
• Authors: Geon Yeong Park, Sangmin Lee, Sang Wan Lee, Jong Chul Ye
• Abstract summary: We present theoretical insight that alerts potential limitations of existing algorithms in exploring unbiased spuriousworks. We then elucidate the importance of bias-conflicting samples on structure learning. Motivated by these observations, we propose a Debiased Contrastive Weight Pruning (DCWP) algorithm, which probes unbiasedworks without expensive group annotations.
• Score: 45.27261440157806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks are often biased to spuriously correlated features that provide misleading statistical evidence that does not generalize. This raises an interesting question: ``Does an optimal unbiased functional subnetwork exist in a severely biased network? If so, how to extract such subnetwork?" While empirical evidence has been accumulated about the existence of such unbiased subnetworks, these observations are mainly based on the guidance of ground-truth unbiased samples. Thus, it is unexplored how to discover the optimal subnetworks with biased training datasets in practice. To address this, here we first present our theoretical insight that alerts potential limitations of existing algorithms in exploring unbiased subnetworks in the presence of strong spurious correlations. We then further elucidate the importance of bias-conflicting samples on structure learning. Motivated by these observations, we propose a Debiased Contrastive Weight Pruning (DCWP) algorithm, which probes unbiased subnetworks without expensive group annotations. Experimental results demonstrate that our approach significantly outperforms state-of-the-art debiasing methods despite its considerable reduction in the number of parameters.

Related papers

• DeNetDM: Debiasing by Network Depth Modulation [6.550893772143]
  We present DeNetDM, a novel debiasing method that uses network depth modulation as a way of developing robustness to spurious correlations. Our method requires no bias annotations or explicit data augmentation while performing on par with approaches that require either or both. We demonstrate that DeNetDM outperforms existing debiasing techniques on both synthetic and real-world datasets by 5%.
  arXiv  Detail & Related papers  (2024-03-28T22:17:19Z)
• Revisiting the Dataset Bias Problem from a Statistical Perspective [72.94990819287551]
  We study the "dataset bias" problem from a statistical standpoint. We identify the main cause of the problem as the strong correlation between a class attribute u and a non-class attribute b. We propose to mitigate dataset bias via either weighting the objective of each sample n by frac1p(u_n|b_n) or sampling that sample with a weight proportional to frac1p(u_n|b_n).
  arXiv  Detail & Related papers  (2024-02-05T22:58:06Z)
• Mining bias-target Alignment from Voronoi Cells [2.66418345185993]
  We propose a bias-agnostic approach to mitigate the impact of bias in deep neural networks. Unlike traditional debiasing approaches, we rely on a metric to quantify bias alignment/misalignment'' on target classes. Our results indicate that the proposed method achieves comparable performance to state-of-the-art supervised approaches.
  arXiv  Detail & Related papers  (2023-05-05T17:09:01Z)
• The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks [64.12052498909105]
  We study the implications of the implicit bias of gradient flow on generalization and adversarial robustness in ReLU networks. In two-layer ReLU networks gradient flow is biased towards solutions that generalize well, but are highly vulnerable to adversarial examples.
  arXiv  Detail & Related papers  (2023-03-02T18:14:35Z)
• Self-supervised debiasing using low rank regularization [59.84695042540525]
  Spurious correlations can cause strong biases in deep neural networks, impairing generalization ability. We propose a self-supervised debiasing framework potentially compatible with unlabeled samples. Remarkably, the proposed debiasing framework significantly improves the generalization performance of self-supervised learning baselines.
  arXiv  Detail & Related papers  (2022-10-11T08:26:19Z)
• Unsupervised Learning of Unbiased Visual Representations [10.871587311621974]
  Deep neural networks are known for their inability to learn robust representations when biases exist in the dataset. We propose a fully unsupervised debiasing framework, consisting of three steps. We employ state-of-the-art supervised debiasing techniques to obtain an unbiased model.
  arXiv  Detail & Related papers  (2022-04-26T10:51:50Z)
• Learning from Failure: Training Debiased Classifier from Biased Classifier [76.52804102765931]
  We show that neural networks learn to rely on spurious correlation only when it is "easier" to learn than the desired knowledge. We propose a failure-based debiasing scheme by training a pair of neural networks simultaneously. Our method significantly improves the training of the network against various types of biases in both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2020-07-06T07:20:29Z)
• Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning [61.93730166203915]
  We argue that traditional notions of fairness are not sufficient when the model is vulnerable to adversarial attacks. We show that measuring robustness bias is a challenging task for DNNs and propose two methods to measure this form of bias.
  arXiv  Detail & Related papers  (2020-06-17T22:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.