Transfer Learning with Uncertainty Quantification: Random Effect
Calibration of Source to Target (RECaST)
- URL: http://arxiv.org/abs/2211.16557v1
- Date: Tue, 29 Nov 2022 19:39:47 GMT
- Title: Transfer Learning with Uncertainty Quantification: Random Effect
Calibration of Source to Target (RECaST)
- Authors: Jimmy Hickey, Jonathan P. Williams, Emily C. Hector
- Abstract summary: We develop a statistical framework for model predictions based on transfer learning, called RECaST.
We mathematically and empirically demonstrate the validity of our RECaST approach for transfer learning between linear models.
We examine our method's performance in a simulation study and in an application to real hospital data.
- Score: 1.8047694351309207
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Transfer learning uses a data model, trained to make predictions or
inferences on data from one population, to make reliable predictions or
inferences on data from another population. Most existing transfer learning
approaches are based on fine-tuning pre-trained neural network models, and fail
to provide crucial uncertainty quantification. We develop a statistical
framework for model predictions based on transfer learning, called RECaST. The
primary mechanism is a Cauchy random effect that recalibrates a source model to
a target population; we mathematically and empirically demonstrate the validity
of our RECaST approach for transfer learning between linear models, in the
sense that prediction sets will achieve their nominal stated coverage, and we
numerically illustrate the method's robustness to asymptotic approximations for
nonlinear models. Whereas many existing techniques are built on particular
source models, RECaST is agnostic to the choice of source model. For example,
our RECaST transfer learning approach can be applied to a continuous or
discrete data model with linear or logistic regression, deep neural network
architectures, etc. Furthermore, RECaST provides uncertainty quantification for
predictions, which is mostly absent in the literature. We examine our method's
performance in a simulation study and in an application to real hospital data.
Related papers
- Transfer Learning for Spatial Autoregressive Models [10.825562180226424]
The spatial autoregressive (SAR) model has been widely applied in various empirical economic studies.
We propose a new transfer learning framework for the SAR model to borrow the information from similar source data to improve both estimation and prediction.
In the empirical application, we apply our method to the election prediction in swing states in the 2020 U.S. presidential election.
arXiv Detail & Related papers (2024-05-20T03:14:15Z) - How Inverse Conditional Flows Can Serve as a Substitute for Distributional Regression [2.9873759776815527]
We propose a framework for distributional regression using inverse flow transformations (DRIFT)
DRIFT covers both interpretable statistical models and flexible neural networks opening up new avenues in both statistical modeling and deep learning.
arXiv Detail & Related papers (2024-05-08T21:19:18Z) - Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
This paper tackles the emerging challenge of training generative models within a self-consuming loop.
We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models.
We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
arXiv Detail & Related papers (2024-02-19T02:08:09Z) - Kalman Filter for Online Classification of Non-Stationary Data [101.26838049872651]
In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps.
We introduce a probabilistic Bayesian online learning model by using a neural representation and a state space model over the linear predictor weights.
In experiments in multi-class classification we demonstrate the predictive ability of the model and its flexibility to capture non-stationarity.
arXiv Detail & Related papers (2023-06-14T11:41:42Z) - Uncertainty estimation for time series forecasting via Gaussian process
regression surrogates [0.8733767481819791]
We propose a new method for uncertainty estimation based on the surrogate Gaussian process model.
Our method can equip any base model with an accurate uncertainty estimate produced by a separate surrogate.
Compared to other approaches, the estimate remains computationally effective with training only one additional model.
arXiv Detail & Related papers (2023-02-06T14:52:56Z) - Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions.
In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data.
We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
arXiv Detail & Related papers (2022-01-11T23:01:12Z) - Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators.
They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions.
We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
arXiv Detail & Related papers (2021-02-22T07:02:37Z) - Transfer Learning for Linear Regression: a Statistical Test of Gain [2.1550839871882017]
Transfer learning aims at reusing knowledge from a source dataset to a similar target one.
It is shown that the quality of transfer for a new input vector $x$ depends on its representation in an eigenbasis.
A statistical test is constructed to predict whether a fine-tuned model has a lower prediction quadratic risk than the base target model.
arXiv Detail & Related papers (2021-02-18T17:46:26Z) - Unlabelled Data Improves Bayesian Uncertainty Calibration under
Covariate Shift [100.52588638477862]
We develop an approximate Bayesian inference scheme based on posterior regularisation.
We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
arXiv Detail & Related papers (2020-06-26T13:50:19Z) - Balance-Subsampled Stable Prediction [55.13512328954456]
We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design.
A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift.
Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
arXiv Detail & Related papers (2020-06-08T07:01:38Z) - A comprehensive study on the prediction reliability of graph neural
networks for virtual screening [0.0]
We investigate the effects of model architectures, regularization methods, and loss functions on the prediction performance and reliability of classification results.
Our result highlights that correct choice of regularization and inference methods is evidently important to achieve high success rate.
arXiv Detail & Related papers (2020-03-17T10:13:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.