OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses

• URL: http://arxiv.org/abs/2204.02426v5
• Date: Mon, 15 Apr 2024 01:11:48 GMT
• Title: OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses
• Authors: Robik Shrestha, Kushal Kafle, Christopher Kanan,
• Abstract summary: We propose modifying the network architecture to impose inductive biases that make the network robust to dataset bias. Specifically, we propose OccamNets, which are biased to favor simpler solutions by design. While OccamNets are biased toward simpler hypotheses, they can learn more complex hypotheses if necessary.
• Score: 24.84797949716142
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dataset bias and spurious correlations can significantly impair generalization in deep neural networks. Many prior efforts have addressed this problem using either alternative loss functions or sampling strategies that focus on rare patterns. We propose a new direction: modifying the network architecture to impose inductive biases that make the network robust to dataset bias. Specifically, we propose OccamNets, which are biased to favor simpler solutions by design. OccamNets have two inductive biases. First, they are biased to use as little network depth as needed for an individual example. Second, they are biased toward using fewer image locations for prediction. While OccamNets are biased toward simpler hypotheses, they can learn more complex hypotheses if necessary. In experiments, OccamNets outperform or rival state-of-the-art methods run on architectures that do not incorporate these inductive biases. Furthermore, we demonstrate that when the state-of-the-art debiasing methods are combined with OccamNets results further improve.

Related papers

• DeNetDM: Debiasing by Network Depth Modulation [6.550893772143]
  We present DeNetDM, a novel debiasing method that uses network depth modulation as a way of developing robustness to spurious correlations. Our method requires no bias annotations or explicit data augmentation while performing on par with approaches that require either or both. We demonstrate that DeNetDM outperforms existing debiasing techniques on both synthetic and real-world datasets by 5%.
  arXiv  Detail & Related papers  (2024-03-28T22:17:19Z)
• Fast Model Debias with Machine Unlearning [54.32026474971696]
  Deep neural networks might behave in a biased manner in many real-world scenarios. Existing debiasing methods suffer from high costs in bias labeling or model re-training. We propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases.
  arXiv  Detail & Related papers  (2023-10-19T08:10:57Z)
• The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks [64.12052498909105]
  We study the implications of the implicit bias of gradient flow on generalization and adversarial robustness in ReLU networks. In two-layer ReLU networks gradient flow is biased towards solutions that generalize well, but are highly vulnerable to adversarial examples.
  arXiv  Detail & Related papers  (2023-03-02T18:14:35Z)
• Self-supervised debiasing using low rank regularization [59.84695042540525]
  Spurious correlations can cause strong biases in deep neural networks, impairing generalization ability. We propose a self-supervised debiasing framework potentially compatible with unlabeled samples. Remarkably, the proposed debiasing framework significantly improves the generalization performance of self-supervised learning baselines.
  arXiv  Detail & Related papers  (2022-10-11T08:26:19Z)
• Training Debiased Subnetworks with Contrastive Weight Pruning [45.27261440157806]
  We present theoretical insight that alerts potential limitations of existing algorithms in exploring unbiased spuriousworks. We then elucidate the importance of bias-conflicting samples on structure learning. Motivated by these observations, we propose a Debiased Contrastive Weight Pruning (DCWP) algorithm, which probes unbiasedworks without expensive group annotations.
  arXiv  Detail & Related papers  (2022-10-11T08:25:47Z)
• Towards Measuring Bias in Image Classification [61.802949761385]
  Convolutional Neural Networks (CNN) have become state-of-the-art for the main computer vision tasks. However, due to the complex structure their decisions are hard to understand which limits their use in some context of the industrial world. We present a systematic approach to uncover data bias by means of attribution maps.
  arXiv  Detail & Related papers  (2021-07-01T10:50:39Z)
• Simon Says: Evaluating and Mitigating Bias in Pruned Neural Networks with Knowledge Distillation [8.238238958749134]
  A clear gap exists in the current literature on evaluating and mitigating bias in pruned neural networks. We propose two simple yet effective metrics, Combined Error Variance (CEV) and Symmetric Distance Error (SDE) to quantitatively evaluate the induced bias prevention quality. Second, we demonstrate that knowledge distillation can mitigate induced bias in pruned neural networks, even with unbalanced datasets. Third, we reveal that model similarity has strong correlations with pruning induced bias, which provides a powerful method to explain why bias occurs in pruned neural networks.
  arXiv  Detail & Related papers  (2021-06-15T02:59:32Z)
• Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization [93.8373619657239]
  Neural networks trained with SGD were recently shown to rely preferentially on linearly-predictive features. This simplicity bias can explain their lack of robustness out of distribution (OOD) We demonstrate that the simplicity bias can be mitigated and OOD generalization improved.
  arXiv  Detail & Related papers  (2021-05-12T12:12:24Z)
• Are Bias Mitigation Techniques for Deep Learning Effective? [24.84797949716142]
  We introduce an improved evaluation protocol, sensible metrics, and a new dataset. We evaluate seven state-of-the-art algorithms using the same network architecture. We find that algorithms exploit hidden biases, are unable to scale to multiple forms of bias, and are highly sensitive to the choice of tuning set.
  arXiv  Detail & Related papers  (2021-04-01T00:14:45Z)
• Learning from Failure: Training Debiased Classifier from Biased Classifier [76.52804102765931]
  We show that neural networks learn to rely on spurious correlation only when it is "easier" to learn than the desired knowledge. We propose a failure-based debiasing scheme by training a pair of neural networks simultaneously. Our method significantly improves the training of the network against various types of biases in both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2020-07-06T07:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.