Estimating the Prediction Performance of Spatial Models via Spatial k-Fold Cross Validation

• URL: http://arxiv.org/abs/2005.14263v1
• Date: Thu, 28 May 2020 19:55:18 GMT
• Title: Estimating the Prediction Performance of Spatial Models via Spatial k-Fold Cross Validation
• Authors: Jonne Pohjankukka, Tapio Pahikkala, Paavo Nevalainen, Jukka Heikkonen
• Abstract summary: In machine learning one often assumes the data are independent when evaluating model performance. spatial autocorrelation (SAC) causes the standard cross validation (CV) methods to produce optimistically biased prediction performance estimates. We propose a modified version of the CV method called spatial k-fold cross validation (SKCV) which provides a useful estimate for model prediction performance without optimistic bias due to SAC.
• Score: 1.7205106391379026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In machine learning one often assumes the data are independent when evaluating model performance. However, this rarely holds in practise. Geographic information data sets are an example where the data points have stronger dependencies among each other the closer they are geographically. This phenomenon known as spatial autocorrelation (SAC) causes the standard cross validation (CV) methods to produce optimistically biased prediction performance estimates for spatial models, which can result in increased costs and accidents in practical applications. To overcome this problem we propose a modified version of the CV method called spatial k-fold cross validation (SKCV), which provides a useful estimate for model prediction performance without optimistic bias due to SAC. We test SKCV with three real world cases involving open natural data showing that the estimates produced by the ordinary CV are up to 40% more optimistic than those of SKCV. Both regression and classification cases are considered in our experiments. In addition, we will show how the SKCV method can be applied as a criterion for selecting data sampling density for new research area.

Related papers

• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting. We validate our theory across a variety of high dimensional data.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional data [7.62566998854384]
  Cross-validation is used for several tasks such as estimating the prediction error, tuning the regularization parameter, and selecting the most suitable predictive model. The K-fold cross-validation is a popular CV method but its limitation is that the risk estimates are highly dependent on the partitioning of the data. This study presents an alternative novel predictive performance test and valid confidence intervals based on exhaustive nested cross-validation.
  arXiv  Detail & Related papers  (2024-08-06T12:28:16Z)
• On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction [1.1470070927586018]
  Cross-validation results are affected by the dissimilarity between the sample data and the prediction locations. We propose a method to quantify such a dissimilarity in the interval 0 to 100%, and from the perspective of the data feature space. Results show that the proposed method can successfully quantify dissimilarity across the entire range of values.
  arXiv  Detail & Related papers  (2024-04-19T01:48:21Z)
• Data-SUITE: Data-centric identification of in-distribution incongruous examples [81.21462458089142]
  Data-SUITE is a data-centric framework to identify incongruous regions of in-distribution (ID) data. We empirically validate Data-SUITE's performance and coverage guarantees.
  arXiv  Detail & Related papers  (2022-02-17T18:58:31Z)
• Confidence intervals for the Cox model test error from cross-validation [91.3755431537592]
  Cross-validation (CV) is one of the most widely used techniques in statistical learning for estimating the test error of a model. Standard confidence intervals for test error using estimates from CV may have coverage below nominal levels. One way to this issue is by estimating the mean squared error of the prediction error instead using nested CV.
  arXiv  Detail & Related papers  (2022-01-26T06:40:43Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Predicting with Confidence on Unseen Distributions [90.68414180153897]
  We connect domain adaptation and predictive uncertainty literature to predict model accuracy on challenging unseen distributions. We find that the difference of confidences (DoC) of a classifier's predictions successfully estimates the classifier's performance change over a variety of shifts. We specifically investigate the distinction between synthetic and natural distribution shifts and observe that despite its simplicity DoC consistently outperforms other quantifications of distributional difference.
  arXiv  Detail & Related papers  (2021-07-07T15:50:18Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Model Rectification via Unknown Unknowns Extraction from Deployment Samples [8.0497115494227]
  We propose a general algorithmic framework that aims to perform a post-training model rectification at deployment time in a supervised way. RTSCV extracts unknown unknowns (u.u.s) We show that RTSCV consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-02-08T11:46:19Z)
• Approximate Cross-validated Mean Estimates for Bayesian Hierarchical Regression Models [6.824747267214373]
  We introduce a novel procedure for obtaining cross-validated predictive estimates for Bayesian hierarchical regression models. We provide theoretical results and demonstrate its efficacy on publicly available data and in simulations.
  arXiv  Detail & Related papers  (2020-11-29T00:00:20Z)
• Approximate Cross-Validation for Structured Models [20.79997929155929]
  Gold standard evaluation technique is structured cross-validation (CV) But CV here can be prohibitively slow due to the need to re-run already-expensive learning algorithms many times. Previous work has shown approximate cross-validation (ACV) methods provide a fast and provably accurate alternative.
  arXiv  Detail & Related papers  (2020-06-23T00:06:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.