A Machine Learning-Ready Data Processing Tool for Near Real-Time Forecasting

• URL: http://arxiv.org/abs/2502.08555v1
• Date: Wed, 12 Feb 2025 16:35:46 GMT
• Title: A Machine Learning-Ready Data Processing Tool for Near Real-Time Forecasting
• Authors: Maher A Dayeh, Michael J Starkey, Subhamoy Chatterjee, Heather Elliott, Samuel Hart, Kimberly Moreland,
• Abstract summary: This paper presents the development of a Machine Learning (ML)- ready data processing tool for Near Real-Time (NRT) space weather forecasting. By merging data from diverse NRT sources, the tool addresses key gaps in current space weather prediction capabilities. The tool processes and structures the data for machine learning models, focusing on time-series forecasting and event detection for extreme solar events.
• Score: 0.0
• License:
• Abstract: Space weather forecasting is critical for mitigating radiation risks in space exploration and protecting Earth-based technologies from geomagnetic disturbances. This paper presents the development of a Machine Learning (ML)- ready data processing tool for Near Real-Time (NRT) space weather forecasting. By merging data from diverse NRT sources such as solar imagery, magnetic field measurements, and energetic particle fluxes, the tool addresses key gaps in current space weather prediction capabilities. The tool processes and structures the data for machine learning models, focusing on time-series forecasting and event detection for extreme solar events. It provides users with a framework to download, process, and label data for ML applications, streamlining the workflow for improved NRT space weather forecasting and scientific research.

Related papers

• Regional Weather Variable Predictions by Machine Learning with Near-Surface Observational and Atmospheric Numerical Data [22.54998659323974]
  This paper presents a novel machine learning (ML) model, called MiMa (short for Micro-Macro), that integrates both near-surface observational data from Kentucky Mesonet stations. MiMa significantly outperforms current models, with Re-MiMa offering precise short-term forecasts for ungauged locations.
  arXiv  Detail & Related papers  (2024-12-11T23:55:38Z)
• Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Data [47.14384085714576]
  We introduce gridded pseudo-tokenPs to handle unstructured observations and a processor containing gridded pseudo-tokens that leverage efficient attention mechanisms. Our method consistently outperforms a range of strong baselines on various synthetic and real-world regression tasks involving large-scale data. The real-life experiments are performed on weather data, demonstrating the potential of our approach to bring performance and computational benefits when applied at scale in a weather modelling pipeline.
  arXiv  Detail & Related papers  (2024-10-09T10:00:56Z)
• Predicting the energetic proton flux with a machine learning regression algorithm [0.0]
  We present a machine learning regression algorithm which is able to forecast the energetic proton flux up to 1 hour ahead. This approach could be helpful to improve monitoring systems of the radiation risk in both deep space and near-Earth environments.
  arXiv  Detail & Related papers  (2024-06-18T15:54:50Z)
• Forecasting SEP Events During Solar Cycles 23 and 24 Using Interpretable Machine Learning [38.321248253111776]
  We employ a suite of machine learning strategies to evaluate the predictive potential of a new data product for a forecast of post-solar flare SEP events. Despite the augmented volume of data, the prediction accuracy reaches 0.7 +- 0.1, which aligns with but does not exceed these published benchmarks.
  arXiv  Detail & Related papers  (2024-03-04T23:12:17Z)
• Observation-Guided Meteorological Field Downscaling at Station Scale: A Benchmark and a New Method [66.80344502790231]
  We extend meteorological downscaling to arbitrary scattered station scales and establish a new benchmark and dataset. Inspired by data assimilation techniques, we integrate observational data into the downscaling process, providing multi-scale observational priors. Our proposed method outperforms other specially designed baseline models on multiple surface variables.
  arXiv  Detail & Related papers  (2024-01-22T14:02:56Z)
• Spatially-resolved hyperlocal weather prediction and anomaly detection using IoT sensor networks and machine learning techniques [0.0]
  We propose a novel approach that combines hyperlocal weather prediction and anomaly detection using IoT sensor networks and machine learning techniques. Our system is able to enhance the spatial resolution of predictions and effectively detect anomalies in real-time. Our findings indicate that this system has the potential to enhance decision-making.
  arXiv  Detail & Related papers  (2023-10-17T05:04:53Z)
• Closing the loop: Autonomous experiments enabled by machine-learning-based online data analysis in synchrotron beamline environments [80.49514665620008]
  Machine learning can be used to enhance research involving large or rapidly generated datasets. In this study, we describe the incorporation of ML into a closed-loop workflow for X-ray reflectometry (XRR) We present solutions that provide an elementary data analysis in real time during the experiment without introducing the additional software dependencies in the beamline control software environment.
  arXiv  Detail & Related papers  (2023-06-20T21:21:19Z)
• Satellite Image Time Series Analysis for Big Earth Observation Data [50.591267188664666]
  This paper describes sits, an open-source R package for satellite image time series analysis using machine learning. We show that this approach produces high accuracy for land use and land cover maps through a case study in the Cerrado biome.
  arXiv  Detail & Related papers  (2022-04-24T15:23:25Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)
• Automated Machine Learning Techniques for Data Streams [91.3755431537592]
  This paper surveys the state-of-the-art open-source AutoML tools, applies them to data collected from streams, and measures how their performance changes over time. The results show that off-the-shelf AutoML tools can provide satisfactory results but in the presence of concept drift, detection or adaptation techniques have to be applied to maintain the predictive accuracy over time.
  arXiv  Detail & Related papers  (2021-06-14T11:42:46Z)
• Toward a Next Generation Particle Precipitation Model: Mesoscale Prediction Through Machine Learning (a Case Study and Framework for Progress) [0.9158190669770423]
  We have compiled, curated, analyzed, and made available a new and more capable database of particle precipitation data. The new total electron energy flux particle precipitation nowcast model, a neural network called PrecipNet, takes advantage of increased expressive power afforded by ML approaches.
  arXiv  Detail & Related papers  (2020-11-19T21:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.