Learning Data-Driven Uncertainty Set Partitions for Robust and Adaptive Energy Forecasting with Missing Data

• URL: http://arxiv.org/abs/2503.20410v1
• Date: Wed, 26 Mar 2025 10:38:56 GMT
• Title: Learning Data-Driven Uncertainty Set Partitions for Robust and Adaptive Energy Forecasting with Missing Data
• Authors: Akylas Stratigakos, Panagiotis Andrianesis,
• Abstract summary: Short-term wind power forecasting models assume the availability of input data (features) when they are deployed and in use.<n>Equipment failures, disruptions, cyberattacks may lead to missing features when such models are used operationally.<n>We use adaptive robust optimization and adversarial machine learning to develop forecasting models that seamlessly handle missing data operationally.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Short-term forecasting models typically assume the availability of input data (features) when they are deployed and in use. However, equipment failures, disruptions, cyberattacks, may lead to missing features when such models are used operationally, which could negatively affect forecast accuracy, and result in suboptimal operational decisions. In this paper, we use adaptive robust optimization and adversarial machine learning to develop forecasting models that seamlessly handle missing data operationally. We propose linear- and neural network-based forecasting models with parameters that adapt to available features, combining linear adaptation with a novel algorithm for learning data-driven uncertainty set partitions. The proposed adaptive models do not rely on identifying historical missing data patterns and are suitable for real-time operations under stringent time constraints. Extensive numerical experiments on short-term wind power forecasting considering horizons from 15 minutes to 4 hours ahead illustrate that our proposed adaptive models are on par with imputation when data are missing for very short periods (e.g., when only the latest measurement is missing) whereas they significantly outperform imputation when data are missing for longer periods. We further provide insights by showcasing how linear adaptation and data-driven partitions (even with a few subsets) approach the performance of the optimal, yet impractical, method of retraining for every possible realization of missing data.

Related papers

• Self-attention-based Diffusion Model for Time-series Imputation in Partial Blackout Scenarios [23.160007389272575]
  Missing values in time series data can harm machine learning performance and introduce bias.<n>Previous work has tackled the imputation of missing data in random, complete blackouts and forecasting scenarios.<n>We introduce a two-stage imputation process using self-attention and diffusion processes to model feature and temporal correlations.
  arXiv  Detail & Related papers  (2025-03-03T16:58:15Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• Certain and Approximately Certain Models for Statistical Learning [4.318959672085627]
  We show that it is possible to learn accurate models directly from data with missing values for certain training data and target models. We build efficient algorithms with theoretical guarantees to check this necessity and return accurate models in cases where imputation is unnecessary.
  arXiv  Detail & Related papers  (2024-02-27T22:49:33Z)
• Enhancing Mean-Reverting Time Series Prediction with Gaussian Processes: Functional and Augmented Data Structures in Financial Forecasting [0.0]
  We explore the application of Gaussian Processes (GPs) for predicting mean-reverting time series with an underlying structure. GPs offer the potential to forecast not just the average prediction but the entire probability distribution over a future trajectory. This is particularly beneficial in financial contexts, where accurate predictions alone may not suffice if incorrect volatility assessments lead to capital losses.
  arXiv  Detail & Related papers  (2024-02-23T06:09:45Z)
• Quilt: Robust Data Segment Selection against Concept Drifts [30.62320149405819]
  Continuous machine learning pipelines are common in industrial settings where models are periodically trained on data streams. concept drifts may occur in data streams where the joint distribution of the data X and label y, P(X, y), changes over time and possibly degrade model accuracy. Existing concept drift adaptation approaches mostly focus on updating the model to the new data and tend to discard the drifted historical data. We propose Quilt, a data-centric framework for identifying and selecting data segments that maximize model accuracy.
  arXiv  Detail & Related papers  (2023-12-15T11:10:34Z)
• Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning [52.021899899683675]
  In scenarios with long-tailed distributions, the model's ability to identify tail classes is limited due to the under-representation of tail samples. We propose an Orthogonal Uncertainty Representation (OUR) of feature embedding and an end-to-end training strategy to improve the long-tail phenomenon of model robustness.
  arXiv  Detail & Related papers  (2023-10-16T05:50:34Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Assessment of Spatio-Temporal Predictors in the Presence of Missing and Heterogeneous Data [23.280400290071732]
  Deep learning approaches achieve outstanding predictive performance in modeling modern data, despite increasing complexity and scale. evaluating the quality of predictive models becomes more challenging as traditional statistical assumptions often no longer hold. This paper introduces a residual analysis framework designed to assess the optimality of temporal-temporal predictive neural models.
  arXiv  Detail & Related papers  (2023-02-03T12:55:08Z)
• DeepVol: Volatility Forecasting from High-Frequency Data with Dilated Causal Convolutions [53.37679435230207]
  We propose DeepVol, a model based on Dilated Causal Convolutions that uses high-frequency data to forecast day-ahead volatility. Our empirical results suggest that the proposed deep learning-based approach effectively learns global features from high-frequency data.
  arXiv  Detail & Related papers  (2022-09-23T16:13:47Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)
• Simple Imputation Rules for Prediction with Missing Data: Contrasting Theoretical Guarantees with Empirical Performance [7.642646077340124]
  Missing data is a common issue in real-world datasets. This paper studies the performance of impute-then-regress pipelines by contrasting theoretical and empirical evidence.
  arXiv  Detail & Related papers  (2021-04-07T14:45:14Z)
• Time-Series Imputation with Wasserstein Interpolation for Optimal Look-Ahead-Bias and Variance Tradeoff [66.59869239999459]
  In finance, imputation of missing returns may be applied prior to training a portfolio optimization model. There is an inherent trade-off between the look-ahead-bias of using the full data set for imputation and the larger variance in the imputation from using only the training data. We propose a Bayesian posterior consensus distribution which optimally controls the variance and look-ahead-bias trade-off in the imputation.
  arXiv  Detail & Related papers  (2021-02-25T09:05:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.