Monitoring Machine Learning Forecasts for Platform Data Streams

• URL: http://arxiv.org/abs/2401.09144v1
• Date: Wed, 17 Jan 2024 11:37:38 GMT
• Title: Monitoring Machine Learning Forecasts for Platform Data Streams
• Authors: Jeroen Rombouts and Ines Wilms
• Abstract summary: Digital platforms require a large-scale forecast framework to respond to sudden performance drops. We propose a data-driven monitoring procedure to answer the question when the ML algorithm should be re-trained. We show that monitor-based re-training produces accurate forecasts compared to viable benchmarks.
• Score: 2.474754293747645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data stream forecasts are essential inputs for decision making at digital platforms. Machine learning algorithms are appealing candidates to produce such forecasts. Yet, digital platforms require a large-scale forecast framework that can flexibly respond to sudden performance drops. Re-training ML algorithms at the same speed as new data batches enter is usually computationally too costly. On the other hand, infrequent re-training requires specifying the re-training frequency and typically comes with a severe cost of forecast deterioration. To ensure accurate and stable forecasts, we propose a simple data-driven monitoring procedure to answer the question when the ML algorithm should be re-trained. Instead of investigating instability of the data streams, we test if the incoming streaming forecast loss batch differs from a well-defined reference batch. Using a novel dataset constituting 15-min frequency data streams from an on-demand logistics platform operating in London, we apply the monitoring procedure to popular ML algorithms including random forest, XGBoost and lasso. We show that monitor-based re-training produces accurate forecasts compared to viable benchmarks while preserving computational feasibility. Moreover, the choice of monitoring procedure is more important than the choice of ML algorithm, thereby permitting practitioners to combine the proposed monitoring procedure with one's favorite forecasting algorithm.

Related papers

• Scaling Laws for Predicting Downstream Performance in LLMs [75.28559015477137]
  This work focuses on the pre-training loss as a more-efficient metric for performance estimation. We extend the power law analytical function to predict domain-specific pre-training loss based on FLOPs across data sources. We employ a two-layer neural network to model the non-linear relationship between multiple domain-specific loss and downstream performance.
  arXiv  Detail & Related papers  (2024-10-11T04:57:48Z)
• Advancing Machine Learning in Industry 4.0: Benchmark Framework for Rare-event Prediction in Chemical Processes [0.0]
  We introduce a novel and comprehensive benchmark framework for rare-event prediction, comparing ML algorithms of varying complexity. We identify optimal ML strategies for predicting abnormal rare events, enabling operators to obtain safer and more reliable plant operations.
  arXiv  Detail & Related papers  (2024-08-31T15:41:10Z)
• Iterative Forgetting: Online Data Stream Regression Using Database-Inspired Adaptive Granulation [1.6874375111244329]
  We present a database-inspired datastream regression model that uses inspiration from R*-trees to create granules from incoming datastreams. Experiments demonstrate that the ability of this method to discard data produces a significant order-of-magnitude improvement in latency and training time.
  arXiv  Detail & Related papers  (2024-03-14T17:26:00Z)
• Direct Unsupervised Denoising [60.71146161035649]
  Unsupervised denoisers do not directly produce a single prediction, such as the MMSE estimate. We present an alternative approach that trains a deterministic network alongside the VAE to directly predict a central tendency.
  arXiv  Detail & Related papers  (2023-10-27T13:02:12Z)
• State Sequences Prediction via Fourier Transform for Representation Learning [111.82376793413746]
  We propose State Sequences Prediction via Fourier Transform (SPF), a novel method for learning expressive representations efficiently. We theoretically analyze the existence of structural information in state sequences, which is closely related to policy performance and signal regularity. Experiments demonstrate that the proposed method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance.
  arXiv  Detail & Related papers  (2023-10-24T14:47:02Z)
• Non-Clairvoyant Scheduling with Predictions Revisited [77.86290991564829]
  In non-clairvoyant scheduling, the task is to find an online strategy for scheduling jobs with a priori unknown processing requirements. We revisit this well-studied problem in a recently popular learning-augmented setting that integrates (untrusted) predictions in algorithm design. We show that these predictions have desired properties, admit a natural error measure as well as algorithms with strong performance guarantees.
  arXiv  Detail & Related papers  (2022-02-21T13:18:11Z)
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting [67.77102283276409]
  We describe two types of forecasting problems: regular grid-based and graph-based. We analyze UQ methods from both the Bayesian and the frequentist point view, casting in a unified framework via statistical decision theory. Through extensive experiments on real-world road network traffic, epidemics, and air quality forecasting tasks, we reveal the statistical computational trade-offs for different UQ methods.
  arXiv  Detail & Related papers  (2021-05-25T14:35:46Z)
• One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning [35.0157090322113]
  Large-scale machine learning systems are often continuously trained with enormous data from production environments. The sheer volume of streaming data poses a significant challenge to real-time training subsystems and ad-hoc sampling is the standard practice. We propose to record a constant amount of information per instance from these forward passes. The extra information measurably improves the selection of which data instances should participate in forward and backward passes.
  arXiv  Detail & Related papers  (2021-04-27T11:29:02Z)
• FlashP: An Analytical Pipeline for Real-time Forecasting of Time-Series Relational Data [31.29499654765994]
  Real-time forecasting can be conducted in two steps: first, we specify the part of data to be focused on and the measure to be predicted by slicing, dicing, and aggregating the data. A natural idea is to utilize sampling to obtain approximate aggregations in real time as the input to train the forecasting model. We introduce a new sampling scheme, called GSW sampling, and analyze error bounds for estimating aggregations using GSW samples.
  arXiv  Detail & Related papers  (2021-01-09T06:23:13Z)
• Online feature selection for rapid, low-overhead learning in networked systems [0.0]
  We present an online algorithm, called OSFS, that selects a small feature set from a large number of available data sources. We find that OSFS requires several hundreds measurements to reduce the number of data sources by two orders of magnitude.
  arXiv  Detail & Related papers  (2020-10-28T12:00:42Z)
• Real-Time Regression with Dividing Local Gaussian Processes [62.01822866877782]
  Local Gaussian processes are a novel, computationally efficient modeling approach based on Gaussian process regression. Due to an iterative, data-driven division of the input space, they achieve a sublinear computational complexity in the total number of training points in practice. A numerical evaluation on real-world data sets shows their advantages over other state-of-the-art methods in terms of accuracy as well as prediction and update speed.
  arXiv  Detail & Related papers  (2020-06-16T18:43:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.