A Lightweight Concept Drift Detection and Adaptation Framework for IoT Data Streams

• URL: http://arxiv.org/abs/2104.10529v1
• Date: Wed, 21 Apr 2021 13:41:41 GMT
• Title: A Lightweight Concept Drift Detection and Adaptation Framework for IoT Data Streams
• Authors: Li Yang, Abdallah Shami
• Abstract summary: We propose an adaptive IoT streaming data analytics framework for anomaly detection use cases based on optimized LightGBM and concept drift adaptation. Experiments on two public datasets show the high accuracy and efficiency of our proposed adaptive LightGBM model.
• Score: 11.411196708408887
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, with the increasing popularity of "Smart Technology", the number of Internet of Things (IoT) devices and systems have surged significantly. Various IoT services and functionalities are based on the analytics of IoT streaming data. However, IoT data analytics faces concept drift challenges due to the dynamic nature of IoT systems and the ever-changing patterns of IoT data streams. In this article, we propose an adaptive IoT streaming data analytics framework for anomaly detection use cases based on optimized LightGBM and concept drift adaptation. A novel drift adaptation method named Optimized Adaptive and Sliding Windowing (OASW) is proposed to adapt to the pattern changes of online IoT data streams. Experiments on two public datasets show the high accuracy and efficiency of our proposed adaptive LightGBM model compared against other state-of-the-art approaches. The proposed adaptive LightGBM model can perform continuous learning and drift adaptation on IoT data streams without human intervention.

Related papers

• AiGAS-dEVL: An Adaptive Incremental Neural Gas Model for Drifting Data Streams under Extreme Verification Latency [6.7236795813629]
  In streaming setups data flows are affected by factors that yield non-stationarities in the patterns (concept drift) We propose a novel approach, AiGAS-dEVL, which relies on growing neural gas to characterize the distributions of all concepts detected within the stream over time. Our approach exposes that the online analysis of the behavior of these points over time facilitates the definition of the evolution of concepts in the feature space.
  arXiv  Detail & Related papers  (2024-07-07T14:04:57Z)
• Automatic AI Model Selection for Wireless Systems: Online Learning via Digital Twinning [50.332027356848094]
  AI-based applications are deployed at intelligent controllers to carry out functionalities like scheduling or power control. The mapping between context and AI model parameters is ideally done in a zero-shot fashion. This paper introduces a general methodology for the online optimization of AMS mappings.
  arXiv  Detail & Related papers  (2024-06-22T11:17:50Z)
• Multi-Stream Cellular Test-Time Adaptation of Real-Time Models Evolving in Dynamic Environments [53.79708667153109]
  Smart objects, notably autonomous vehicles, face challenges in critical local computations due to limited resources. We propose a novel Multi-Stream Cellular Test-Time Adaptation setup where models adapt on the fly to a dynamic environment divided into cells. We validate our methodology in the context of autonomous vehicles navigating across cells defined based on location and weather conditions.
  arXiv  Detail & Related papers  (2024-04-27T15:00:57Z)
• Effective Intrusion Detection in Heterogeneous Internet-of-Things Networks via Ensemble Knowledge Distillation-based Federated Learning [52.6706505729803]
  We introduce Federated Learning (FL) to collaboratively train a decentralized shared model of Intrusion Detection Systems (IDS) FLEKD enables a more flexible aggregation method than conventional model fusion techniques. Experiment results show that the proposed approach outperforms local training and traditional FL in terms of both speed and performance.
  arXiv  Detail & Related papers  (2024-01-22T14:16:37Z)
• Federated Deep Learning for Intrusion Detection in IoT Networks [1.3097853961043058]
  A common approach to implementing AI-based Intrusion Detection systems (IDSs) in distributed IoT systems is in a centralised manner. This approach may violate data privacy and prohibit IDS scalability. We design an experiment representative of the real world and evaluate the performance of an FL-based IDS.
  arXiv  Detail & Related papers  (2023-06-05T09:08:24Z)
• IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective [10.350337750192997]
  We conduct a review of existing methods in the model selection, tuning, and updating procedures in the area of AutoML. To justify our findings and help industrial users and researchers better implement AutoML approaches, a case study of applying AutoML to IoT anomaly detection problems is conducted.
  arXiv  Detail & Related papers  (2022-09-16T16:02:56Z)
• Bayesian Optimization and Deep Learning forsteering wheel angle prediction [58.720142291102135]
  This work aims to obtain an accurate model for the prediction of the steering angle in an automated driving system. BO was able to identify, within a limited number of trials, a model -- namely BOST-LSTM -- which resulted, the most accurate when compared to classical end-to-end driving models.
  arXiv  Detail & Related papers  (2021-10-22T15:25:14Z)
• PWPAE: An Ensemble Framework for Concept Drift Adaptation in IoT Data Streams [9.953967527396316]
  We propose a Performance Weighted Probability Averaging Ensemble (PWPAE) framework for drift adaptive IoT anomaly detection through IoT data stream analytics. Experiments on two public datasets show the effectiveness of our proposed PWPAE method compared against state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-09-10T17:50:49Z)
• Adaptive Anomaly Detection for Internet of Things in Hierarchical Edge Computing: A Contextual-Bandit Approach [81.5261621619557]
  We propose an adaptive anomaly detection scheme with hierarchical edge computing (HEC) We first construct multiple anomaly detection DNN models with increasing complexity, and associate each of them to a corresponding HEC layer. Then, we design an adaptive model selection scheme that is formulated as a contextual-bandit problem and solved by using a reinforcement learning policy network.
  arXiv  Detail & Related papers  (2021-08-09T08:45:47Z)
• Contextual-Bandit Anomaly Detection for IoT Data in Distributed Hierarchical Edge Computing [65.78881372074983]
  IoT devices can hardly afford complex deep neural networks (DNN) models, and offloading anomaly detection tasks to the cloud incurs long delay. We propose and build a demo for an adaptive anomaly detection approach for distributed hierarchical edge computing (HEC) systems. We show that our proposed approach significantly reduces detection delay without sacrificing accuracy, as compared to offloading detection tasks to the cloud.
  arXiv  Detail & Related papers  (2020-04-15T06:13:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.