Supervised learning from noisy observations: Combining machine-learning techniques with data assimilation

• URL: http://arxiv.org/abs/2007.07383v3
• Date: Mon, 8 Mar 2021 11:45:38 GMT
• Title: Supervised learning from noisy observations: Combining machine-learning techniques with data assimilation
• Authors: Georg A. Gottwald and Sebastian Reich
• Abstract summary: We show how to optimally combine forecast models and their inherent uncertainty with incoming noisy observations. We show that the obtained forecast model has remarkably good forecast skill while being computationally cheap once trained. Going beyond the task of forecasting, we show that our method can be used to generate reliable ensembles for probabilistic forecasting as well as to learn effective model closure in multi-scale systems.
• Score: 0.6091702876917281
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data-driven prediction and physics-agnostic machine-learning methods have attracted increased interest in recent years achieving forecast horizons going well beyond those to be expected for chaotic dynamical systems. In a separate strand of research data-assimilation has been successfully used to optimally combine forecast models and their inherent uncertainty with incoming noisy observations. The key idea in our work here is to achieve increased forecast capabilities by judiciously combining machine-learning algorithms and data assimilation. We combine the physics-agnostic data-driven approach of random feature maps as a forecast model within an ensemble Kalman filter data assimilation procedure. The machine-learning model is learned sequentially by incorporating incoming noisy observations. We show that the obtained forecast model has remarkably good forecast skill while being computationally cheap once trained. Going beyond the task of forecasting, we show that our method can be used to generate reliable ensembles for probabilistic forecasting as well as to learn effective model closure in multi-scale systems.

Related papers

• Attribute-to-Delete: Machine Unlearning via Datamodel Matching [65.13151619119782]
  Machine unlearning -- efficiently removing a small "forget set" training data on a pre-divertrained machine learning model -- has recently attracted interest. Recent research shows that machine unlearning techniques do not hold up in such a challenging setting.
  arXiv  Detail & Related papers  (2024-10-30T17:20:10Z)
• FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation [67.20588721130623]
  We develop an AI-based cyclic weather forecasting system, FengWu-4DVar. FengWu-4DVar can incorporate observational data into the data-driven weather forecasting model. Experiments on the simulated observational dataset demonstrate that FengWu-4DVar is capable of generating reasonable analysis fields.
  arXiv  Detail & Related papers  (2023-12-16T02:07:56Z)
• EAMDrift: An interpretable self retrain model for time series [0.0]
  We present EAMDrift, a novel method that combines forecasts from multiple individual predictors by weighting each prediction according to a performance metric. EAMDrift is designed to automatically adapt to out-of-distribution patterns in data and identify the most appropriate models to use at each moment. Our study on real-world datasets shows that EAMDrift outperforms individual baseline models by 20% and achieves comparable accuracy results to non-interpretable ensemble models.
  arXiv  Detail & Related papers  (2023-05-31T13:25:26Z)
• Online machine-learning forecast uncertainty estimation for sequential data assimilation [0.0]
  Quantifying forecast uncertainty is a key aspect of state-of-the-art numerical weather prediction and data assimilation systems. In this work a machine learning method is presented based on convolutional neural networks that estimates the state-dependent forecast uncertainty. The hybrid data assimilation method shows similar performance to the ensemble Kalman filter outperforming it when the ensembles are relatively small.
  arXiv  Detail & Related papers  (2023-05-12T19:23:21Z)
• Prediction-Powered Inference [68.97619568620709]
  Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning.
  arXiv  Detail & Related papers  (2023-01-23T18:59:28Z)
• On Learning Prediction-Focused Mixtures [30.338543175315507]
  We introduce prediction-focused modeling for mixtures, which automatically selects the dimensions relevant to the prediction task. Our approach identifies relevant signal from the input, outperforms models that are not prediction-focused, and is easy to optimize.
  arXiv  Detail & Related papers  (2021-10-25T19:14:36Z)
• A Meta-learning Approach to Reservoir Computing: Time Series Prediction with Limited Data [0.0]
  We present a data-driven approach to automatically extract an appropriate model structure from experimentally observed processes. We demonstrate our approach on a simple benchmark problem, where it beats the state of the art meta-learning techniques.
  arXiv  Detail & Related papers  (2021-10-07T18:23:14Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
  We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
  arXiv  Detail & Related papers  (2021-02-15T19:56:48Z)
• Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning [7.194382512848327]
  We propose a new parametrization to supervised learning on state-action data to stably predict at longer horizons. Our results in simulated and experimental robotic tasks show that our trajectory-based models yield significantly more accurate long term predictions.
  arXiv  Detail & Related papers  (2020-12-16T18:47:37Z)
• Video Prediction via Example Guidance [156.08546987158616]
  In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that can efficiently predict plausible future states.
  arXiv  Detail & Related papers  (2020-07-03T14:57:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.