Understanding Model Drift in a Large Cellular Network
- URL: http://arxiv.org/abs/2109.03011v1
- Date: Tue, 7 Sep 2021 11:57:07 GMT
- Title: Understanding Model Drift in a Large Cellular Network
- Authors: Shinan Liu, Francesco Bronzino, Paul Schmitt, Nick Feamster, Ricardo
Borges, Hector Garcia Crespo, Brian Ward
- Abstract summary: Concept drift can degrade accuracy of machine learning models over time.
This paper explores concept drift in a large cellular network in the United States for a major metropolitan area in the context of demand forecasting.
We find that concept drift arises largely due to data drift, and it appears across different key performance indicators (KPIs), models, training set sizes, and time intervals.
- Score: 9.323853587087457
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Operational networks are increasingly using machine learning models for a
variety of tasks, including detecting anomalies, inferring application
performance, and forecasting demand. Accurate models are important, yet
accuracy can degrade over time due to concept drift, whereby either the
characteristics of the data change over time (data drift) or the relationship
between the features and the target predictor change over time (model drift).
Drift is important to detect because changes in properties of the underlying
data or relationships to the target prediction can require model retraining,
which can be time-consuming and expensive. Concept drift occurs in operational
networks for a variety of reasons, ranging from software upgrades to
seasonality to changes in user behavior. Yet, despite the prevalence of drift
in networks, its extent and effects on prediction accuracy have not been
extensively studied. This paper presents an initial exploration into concept
drift in a large cellular network in the United States for a major metropolitan
area in the context of demand forecasting. We find that concept drift arises
largely due to data drift, and it appears across different key performance
indicators (KPIs), models, training set sizes, and time intervals. We identify
the sources of concept drift for the particular problem of forecasting downlink
volume. Weekly and seasonal patterns introduce both high and low-frequency
model drift, while disasters and upgrades result in sudden drift due to
exogenous shocks. Regions with high population density, lower traffic volumes,
and higher speeds also tend to correlate with more concept drift. The features
that contribute most significantly to concept drift are User Equipment (UE)
downlink packets, UE uplink packets, and Real-time Transport Protocol (RTP)
total received packets.
Related papers
- datadriftR: An R Package for Concept Drift Detection in Predictive Models [0.0]
This paper introduces drifter, an R package designed to detect concept drift.
It proposes a novel method called Profile Drift Detection (PDD) that enables both drift detection and an enhanced understanding of the cause behind the drift.
arXiv Detail & Related papers (2024-12-15T20:59:49Z) - Adapting to Length Shift: FlexiLength Network for Trajectory Prediction [53.637837706712794]
Trajectory prediction plays an important role in various applications, including autonomous driving, robotics, and scene understanding.
Existing approaches mainly focus on developing compact neural networks to increase prediction precision on public datasets, typically employing a standardized input duration.
We introduce a general and effective framework, the FlexiLength Network (FLN), to enhance the robustness of existing trajectory prediction against varying observation periods.
arXiv Detail & Related papers (2024-03-31T17:18:57Z) - Explaining Drift using Shapley Values [0.0]
Machine learning models often deteriorate in their performance when they are used to predict the outcomes over data on which they were not trained.
There is no framework to identify the drivers behind the drift in model performance.
We propose a novel framework - DBShap that uses principled Shapley values to identify the main contributors of the drift.
arXiv Detail & Related papers (2024-01-18T07:07:42Z) - MemDA: Forecasting Urban Time Series with Memory-based Drift Adaptation [24.284969264008733]
We propose a new urban time series prediction model for the concept drift problem, which encodes the drift by considering the periodicity in the data.
Our design significantly outperforms state-of-the-art methods and can be well generalized to existing prediction backbones.
arXiv Detail & Related papers (2023-09-25T15:22:28Z) - FLARE: Detection and Mitigation of Concept Drift for Federated Learning
based IoT Deployments [2.7776688429637466]
FLARE is a lightweight dual-scheduler FL framework that conditionally transfers training data and deploys models between edge and sensor endpoints.
We show that FLARE can significantly reduce the amount of data exchanged between edge and sensor nodes compared to fixed-interval scheduling methods.
It can successfully detect concept drift reactively with at least a 16x reduction in latency.
arXiv Detail & Related papers (2023-05-15T10:09:07Z) - Correlating sparse sensing for large-scale traffic speed estimation: A
Laplacian-enhanced low-rank tensor kriging approach [76.45949280328838]
We propose a Laplacian enhanced low-rank tensor (LETC) framework featuring both lowrankness and multi-temporal correlations for large-scale traffic speed kriging.
We then design an efficient solution algorithm via several effective numeric techniques to scale up the proposed model to network-wide kriging.
arXiv Detail & Related papers (2022-10-21T07:25:57Z) - Real-time Object Detection for Streaming Perception [84.2559631820007]
Streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception.
We build a simple and effective framework for streaming perception.
Our method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline.
arXiv Detail & Related papers (2022-03-23T11:33:27Z) - Detecting Concept Drift With Neural Network Model Uncertainty [0.0]
Uncertainty Drift Detection (UDD) is able to detect drifts without access to true labels.
In contrast to input data-based drift detection, our approach considers the effects of the current input data on the properties of the prediction model.
We show that UDD outperforms other state-of-the-art strategies on two synthetic as well as ten real-world data sets for both regression and classification tasks.
arXiv Detail & Related papers (2021-07-05T08:56:36Z) - Predicting traffic signals on transportation networks using
spatio-temporal correlations on graphs [56.48498624951417]
This paper proposes a traffic propagation model that merges multiple heat diffusion kernels into a data-driven prediction model to forecast traffic signals.
We optimize the model parameters using Bayesian inference to minimize the prediction errors and, consequently, determine the mixing ratio of the two approaches.
The proposed model demonstrates prediction accuracy comparable to that of the state-of-the-art deep neural networks with lower computational effort.
arXiv Detail & Related papers (2021-04-27T18:17:42Z) - Switching Scheme: A Novel Approach for Handling Incremental Concept
Drift in Real-World Data Sets [0.0]
Concept drifts can severely affect the prediction performance of a machine learning system.
In this work, we analyze the effects of concept drifts in the context of a real-world data set.
We introduce the switching scheme which combines the two principles of retraining and updating of a machine learning model.
arXiv Detail & Related papers (2020-11-05T10:16:54Z) - On Robustness and Transferability of Convolutional Neural Networks [147.71743081671508]
Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts.
We study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time.
We find that increasing both the training set and model sizes significantly improve the distributional shift robustness.
arXiv Detail & Related papers (2020-07-16T18:39:04Z) - Spatial-Temporal Transformer Networks for Traffic Flow Forecasting [74.76852538940746]
We propose a novel paradigm of Spatial-Temporal Transformer Networks (STTNs) to improve the accuracy of long-term traffic forecasting.
Specifically, we present a new variant of graph neural networks, named spatial transformer, by dynamically modeling directed spatial dependencies.
The proposed model enables fast and scalable training over a long range spatial-temporal dependencies.
arXiv Detail & Related papers (2020-01-09T10:21:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.