Out-of-Distribution Generalization Analysis via Influence Function

• URL: http://arxiv.org/abs/2101.08521v1
• Date: Thu, 21 Jan 2021 09:59:55 GMT
• Title: Out-of-Distribution Generalization Analysis via Influence Function
• Authors: Haotian Ye, Chuanlong Xie, Yue Liu, Zhenguo Li
• Abstract summary: The mismatch between training and target data is one major challenge for machine learning systems. We introduce Influence Function, a classical tool from robust statistics, into the OOD generalization problem. We show that the accuracy on test domains and the proposed index together can help us discern whether OOD algorithms are needed and whether a model achieves good OOD generalization.
• Score: 25.80365416547478
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The mismatch between training and target data is one major challenge for current machine learning systems. When training data is collected from multiple domains and the target domains include all training domains and other new domains, we are facing an Out-of-Distribution (OOD) generalization problem that aims to find a model with the best OOD accuracy. One of the definitions of OOD accuracy is worst-domain accuracy. In general, the set of target domains is unknown, and the worst over target domains may be unseen when the number of observed domains is limited. In this paper, we show that the worst accuracy over the observed domains may dramatically fail to identify the OOD accuracy. To this end, we introduce Influence Function, a classical tool from robust statistics, into the OOD generalization problem and suggest the variance of influence function to monitor the stability of a model on training domains. We show that the accuracy on test domains and the proposed index together can help us discern whether OOD algorithms are needed and whether a model achieves good OOD generalization.

Related papers

• MADOD: Generalizing OOD Detection to Unseen Domains via G-Invariance Meta-Learning [10.38552112657656]
  We introduce Meta-learned Across Domain Out-of-distribution Detection (MADOD), a novel framework designed to address both shifts concurrently. Our key innovation lies in task construction: we randomly designate in-distribution classes as pseudo-OODs within each meta-learning task. Experiments on real-world and synthetic datasets demonstrate MADOD's superior performance in semantic OOD detection across unseen domains.
  arXiv  Detail & Related papers  (2024-11-02T17:46:23Z)
• Domain penalisation for improved Out-of-Distribution Generalisation [1.979158763744267]
  Domain generalisation (DG) aims to ensure robust performance across diverse and unseen target domains. We propose a framework for the task of object detection, where the data is assumed to be sampled from multiple source domains. By prioritising the domains that needs more attention, our approach effectively balances the training process.
  arXiv  Detail & Related papers  (2024-08-03T11:06:47Z)
• Consistency-Guided Temperature Scaling Using Style and Content Information for Out-of-Domain Calibration [24.89907794192497]
  We propose consistency-guided temperature scaling (CTS) to enhance out-of-domain calibration performance. We take consistencies into account in terms of two different aspects -- style and content -- which are the key components that can well-represent data samples in multi-domain settings. This can be accomplished by employing only the source domains without compromising accuracy, making our scheme directly applicable to various trustworthy AI systems.
  arXiv  Detail & Related papers  (2024-02-22T23:36:18Z)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models [97.19901765814431]
  This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. We show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value.
  arXiv  Detail & Related papers  (2023-11-03T05:41:25Z)
• Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection [28.810524375810736]
  Out-of-distribution (OOD) detection is a critical task for reliable predictions over text. Fine-tuning with pre-trained language models has been a de facto procedure to derive OOD detectors. We show that using distance-based detection methods, pre-trained language models are near-perfect OOD detectors when the distribution shift involves a domain change.
  arXiv  Detail & Related papers  (2023-05-22T17:42:44Z)
• Out-of-Domain Robustness via Targeted Augmentations [90.94290420322457]
  We study principles for designing data augmentations for out-of-domain generalization. Motivated by theoretical analysis on a linear setting, we propose targeted augmentations. We show that targeted augmentations set new states-of-the-art for OOD performance by 3.2-15.2 percentage points.
  arXiv  Detail & Related papers  (2023-02-23T08:59:56Z)
• Generalizability of Adversarial Robustness Under Distribution Shifts [57.767152566761304]
  We take a first step towards investigating the interplay between empirical and certified adversarial robustness on one hand and domain generalization on another. We train robust models on multiple domains and evaluate their accuracy and robustness on an unseen domain. We extend our study to cover a real-world medical application, in which adversarial augmentation significantly boosts the generalization of robustness with minimal effect on clean data accuracy.
  arXiv  Detail & Related papers  (2022-09-29T18:25:48Z)
• Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives [16.68091981866261]
  Unsupervised domain adaptation (UDA) is proposed to counter the performance drop on data in a target domain. UDA has yielded promising results on natural image processing, video analysis, natural language processing, time-series data analysis, medical image analysis, etc.
  arXiv  Detail & Related papers  (2022-08-15T20:05:07Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Self-balanced Learning For Domain Generalization [64.99791119112503]
  Domain generalization aims to learn a prediction model on multi-domain source data such that the model can generalize to a target domain with unknown statistics. Most existing approaches have been developed under the assumption that the source data is well-balanced in terms of both domain and class. We propose a self-balanced domain generalization framework that adaptively learns the weights of losses to alleviate the bias caused by different distributions of the multi-domain source data.
  arXiv  Detail & Related papers  (2021-08-31T03:17:54Z)
• Adaptive Risk Minimization: Learning to Adapt to Domain Shift [109.87561509436016]
  A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution. In this work, we consider the problem setting of domain generalization, where the training data are structured into domains and there may be multiple test time shifts. We introduce the framework of adaptive risk minimization (ARM), in which models are directly optimized for effective adaptation to shift by learning to adapt on the training domains.
  arXiv  Detail & Related papers  (2020-07-06T17:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.