Avoiding spurious correlations via logit correction
- URL: http://arxiv.org/abs/2212.01433v1
- Date: Fri, 2 Dec 2022 20:30:59 GMT
- Title: Avoiding spurious correlations via logit correction
- Authors: Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, Carlos
Fernandez-Granda
- Abstract summary: Empirical studies suggest that machine learning models trained with empirical risk often rely on attributes that may be spuriously correlated with the class labels.
In this work, we consider a situation where potential spurious correlations are present in the majority of training data.
We propose the logit correction (LC) loss, a simple yet effective improvement on the softmax cross-entropy loss, to correct the sample logit.
- Score: 21.261525854506743
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Empirical studies suggest that machine learning models trained with empirical
risk minimization (ERM) often rely on attributes that may be spuriously
correlated with the class labels. Such models typically lead to poor
performance during inference for data lacking such correlations. In this work,
we explicitly consider a situation where potential spurious correlations are
present in the majority of training data. In contrast with existing approaches,
which use the ERM model outputs to detect the samples without spurious
correlations, and either heuristically upweighting or upsampling those samples;
we propose the logit correction (LC) loss, a simple yet effective improvement
on the softmax cross-entropy loss, to correct the sample logit. We demonstrate
that minimizing the LC loss is equivalent to maximizing the group-balanced
accuracy, so the proposed LC could mitigate the negative impacts of spurious
correlations. Our extensive experimental results further reveal that the
proposed LC loss outperforms the SoTA solutions on multiple popular benchmarks
by a large margin, an average 5.5% absolute improvement, without access to
spurious attribute labels. LC is also competitive with oracle methods that make
use of the attribute labels. Code is available at
https://github.com/shengliu66/LC.
Related papers
- Trained Models Tell Us How to Make Them Robust to Spurious Correlation without Group Annotation [3.894771553698554]
Empirical Risk Minimization (ERM) models tend to rely on attributes that have high spurious correlation with the target.
This can degrade the performance on underrepresented (or'minority') groups that lack these attributes.
We propose Environment-based Validation and Loss-based Sampling (EVaLS) to enhance robustness to spurious correlation.
arXiv Detail & Related papers (2024-10-07T08:17:44Z) - Learning with Imbalanced Noisy Data by Preventing Bias in Sample
Selection [82.43311784594384]
Real-world datasets contain not only noisy labels but also class imbalance.
We propose a simple yet effective method to address noisy labels in imbalanced datasets.
arXiv Detail & Related papers (2024-02-17T10:34:53Z) - Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice.
We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
arXiv Detail & Related papers (2023-12-27T09:03:43Z) - Mixed Pseudo Labels for Semi-Supervised Object Detection [27.735659283870646]
This paper proposes Mixed Pseudo Labels (MixPL), consisting of Mixup and Mosaic for pseudo-labeled data, to mitigate the negative impact of missed detections.
MixPL consistently improves the performance of various detectors and obtains new state-of-the-art results with Faster R-CNN, FCOS, and DINO on COCO-Standard and COCO-Full benchmarks.
arXiv Detail & Related papers (2023-12-12T06:35:27Z) - Annotation-Free Group Robustness via Loss-Based Resampling [3.355491272942994]
Training neural networks for image classification with empirical risk minimization makes them vulnerable to relying on spurious attributes instead of causal ones for prediction.
We propose a new method, called loss-based feature re-weighting (LFR), in which we infer a grouping of the data by evaluating an ERM-pre-trained model on a small left-out split of the training data.
For a complete assessment, we evaluate LFR on various versions of Waterbirds and CelebA datasets with different spurious correlations.
arXiv Detail & Related papers (2023-12-08T08:22:02Z) - Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data.
We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
arXiv Detail & Related papers (2023-03-06T14:49:59Z) - Learning with Noisy Labels over Imbalanced Subpopulations [13.477553187049462]
Learning with noisy labels (LNL) has attracted significant attention from the research community.
We propose a novel LNL method to simultaneously deal with noisy labels and imbalanced subpopulations.
We introduce a feature-based metric that takes the sample correlation into account for estimating samples' clean probabilities.
arXiv Detail & Related papers (2022-11-16T07:25:24Z) - Hierarchical Semi-Supervised Contrastive Learning for
Contamination-Resistant Anomaly Detection [81.07346419422605]
Anomaly detection aims at identifying deviant samples from the normal data distribution.
Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies.
We propose a novel hierarchical semi-supervised contrastive learning framework, for contamination-resistant anomaly detection.
arXiv Detail & Related papers (2022-07-24T18:49:26Z) - Correct-N-Contrast: A Contrastive Approach for Improving Robustness to
Spurious Correlations [59.24031936150582]
Spurious correlations pose a major challenge for robust machine learning.
Models trained with empirical risk minimization (ERM) may learn to rely on correlations between class labels and spurious attributes.
We propose Correct-N-Contrast (CNC), a contrastive approach to directly learn representations robust to spurious correlations.
arXiv Detail & Related papers (2022-03-03T05:03:28Z) - Risk Minimization from Adaptively Collected Data: Guarantees for
Supervised and Policy Learning [57.88785630755165]
Empirical risk minimization (ERM) is the workhorse of machine learning, but its model-agnostic guarantees can fail when we use adaptively collected data.
We study a generic importance sampling weighted ERM algorithm for using adaptively collected data to minimize the average of a loss function over a hypothesis class.
For policy learning, we provide rate-optimal regret guarantees that close an open gap in the existing literature whenever exploration decays to zero.
arXiv Detail & Related papers (2021-06-03T09:50:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.