Data Leakage Inflates Generalizability of Power Outage Prediction Models

• URL: http://arxiv.org/abs/2608.24665v1
• Date: Tue, 25 Aug 2026 15:05:55 GMT
• Title: Data Leakage Inflates Generalizability of Power Outage Prediction Models
• Abstract summary: We identify three common methodological choices in power outage prediction models that influence their ability to generalize.<n>We compare the predictive performance impacts of different methodological decisions using publicly available data for the U.S. East Coast from 2018 to 2023.<n>While random train-test splits yield strong performance, we show that these results are inflated by spatial and temporal autocorrelation.<n>These findings suggest that, given current data availability and evaluation practices, publicly trained outage prediction models offer limited and uncertain operational value.
• Score: 0.9558392439655014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Power outage prediction models are increasingly used in assessments of climate-driven infrastructure risk, yet current evaluation practices obscure whether these models generalize to the novel conditions such applications require. We identify three common methodological choices in power outage prediction models that influence their ability to generalize across spatial, temporal, and event-based settings. We compare the predictive performance impacts of different methodological decisions using publicly available data for the U.S. East Coast from 2018 to 2023 and feature sets derived from weather reanalysis and land-cover data, and embeddings from a GeoAI foundation model (Prithvi WxC). Specifically, we assess model performance under multiple test selection strategies, including unfiltered random splits, leave-one-state-out, and leave-one-event-out designs, which increasingly approximate real-world deployment conditions. While random train-test splits yield strong performance, we show that these results are inflated by spatial and temporal autocorrelation. Under spatial and temporal holdout experiments, predictive accuracy degrades substantially, with models often failing to outperform a simple null baseline. Incorporating GeoAI foundation model embeddings yields limited and inconsistent improvements, primarily for spatial generalization, and does not resolve poor event-level transferability. These findings suggest that, given current data availability and evaluation practices, publicly trained outage prediction models offer limited and uncertain operational value. Progress will likely require improved data coverage, more realistic evaluation protocols, and a shift in focus from marginal modeling advances toward addressing structural data constraints.

Related papers

• Precipitation Downscaling Using Foundation Model-Conditioned Diffusion [1.1848987098427715]
  High-resolution precipitation fields are essential for hydrological impact assessment, yet global climate model outputs are too coarse and biased for direct use.<n>We investigate three conditioning strategies for a denoising diffusion probabilistic model applied to daily precipitation downscaling.<n>Concatenation conditioning achieves the lowest point-wise CRPS and MSE, but tends to produce over-smoothed fields that suppress high-intensity events.<n>Cross-attention conditioning provides substantially better distributional realism and modest improvements in spectral fidelity.
  arXiv  Detail & Related papers  (2026-08-26T14:30:39Z)
• Why Model Selection Fails in Time Series Forecasting: An Empirical Study of Instability Across Data Regimes [0.0]
  Time series forecasting models often exhibit inconsistent performance across datasets with varying statistical and structural properties.<n>This paper investigates why rule-based model selection fails in time series forecasting by analyzing the relationship between data-regime descriptors and model performance.
  arXiv  Detail & Related papers  (2026-05-02T21:19:29Z)
• Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts [2.4329932351753114]
  "No-analog" future climate states are regimes where external forcing drives the system into conditions outside the empirical range of the historical training data.<n>We benchmark the robustness of three state-of-the-art architectures: U-Net, ConvLSTM, and the ClimaX foundation model specifically restricted to a historical-only training regime.
  arXiv  Detail & Related papers  (2026-03-24T10:29:54Z)
• Target Concept Tuning Improves Extreme Weather Forecasting [51.43578341469291]
  Deep learning models for meteorological forecasting often fail in rare but high-impact events such as typhoons.<n>Existing fine-tuning methods face a trade-off between overlooking these extreme events and overfitting them at the expense of overall performance.<n>We propose TaCT, an interpretable concept-gated fine-tuning framework that solves the aforementioned issue by selective model improvement.
  arXiv  Detail & Related papers  (2026-03-17T15:34:05Z)
• Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving [54.46325690390831]
  We propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment.<n>MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine.<n>MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes.
  arXiv  Detail & Related papers  (2025-11-26T17:01:41Z)
• On conditional diffusion models for PDE simulations [53.01911265639582]
  We study score-based diffusion models for forecasting and assimilation of sparse observations. We propose an autoregressive sampling approach that significantly improves performance in forecasting. We also propose a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths.
  arXiv  Detail & Related papers  (2024-10-21T18:31:04Z)
• Physics-guided Active Sample Reweighting for Urban Flow Prediction [75.24539704456791]
  Urban flow prediction is a nuanced-temporal modeling that estimates the throughput of transportation services like buses, taxis and ride-driven models. Some recent prediction solutions bring remedies with the notion of physics-guided machine learning (PGML) We develop a atized physics-guided network (PN), and propose a data-aware framework Physics-guided Active Sample Reweighting (P-GASR)
  arXiv  Detail & Related papers  (2024-07-18T15:44:23Z)
• Inference at the data's edge: Gaussian processes for modeling and inference under model-dependency, poor overlap, and extrapolation [0.0]
  The Gaussian Process (GP) is a flexible non-linear regression approach. It provides a principled approach to handling our uncertainty over predicted (counterfactual) values. This is especially valuable under conditions of extrapolation or weak overlap.
  arXiv  Detail & Related papers  (2024-07-15T05:09:50Z)
• A Temporally Disentangled Contrastive Diffusion Model for Spatiotemporal Imputation [35.46631415365955]
  We introduce a conditional diffusion framework called C$2$TSD, which incorporates disentangled temporal (trend and seasonality) representations as conditional information. Our experiments on three real-world datasets demonstrate the superior performance of our approach compared to a number of state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-18T11:59:04Z)
• ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast [57.6987191099507]
  We introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast. We also introduce ExBooster, which captures the uncertainty in prediction outcomes by employing multiple random samples. Our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.
  arXiv  Detail & Related papers  (2024-02-02T10:34:13Z)
• Long-term drought prediction using deep neural networks based on geospatial weather data [75.38539438000072]
  High-quality drought forecasting up to a year in advance is critical for agriculture planning and insurance. We tackle drought data by introducing an end-to-end approach that adopts a systematic end-to-end approach. Key findings are the exceptional performance of a Transformer model, EarthFormer, in making accurate short-term (up to six months) forecasts.
  arXiv  Detail & Related papers  (2023-09-12T13:28:06Z)
• Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future [73.03458424369657]
  In real-time forecasting in public health, data collection is a non-trivial and demanding task. 'Backfill' phenomenon and its effect on model performance has been barely studied in the prior literature. We formulate a novel problem and neural framework Back2Future that aims to refine a given model's predictions in real-time.
  arXiv  Detail & Related papers  (2021-06-08T14:48:20Z)
• Learning Prediction Intervals for Model Performance [1.433758865948252]
  We propose a method to compute prediction intervals for model performance. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines.
  arXiv  Detail & Related papers  (2020-12-15T21:32:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.