An Automated Data Engineering Pipeline for Anomaly Detection of IoT Sensor Data

• URL: http://arxiv.org/abs/2109.13828v1
• Date: Tue, 28 Sep 2021 15:57:29 GMT
• Title: An Automated Data Engineering Pipeline for Anomaly Detection of IoT Sensor Data
• Authors: Xinze Li, Baixi Zou
• Abstract summary: System of Chip technology, Internet of Things (IoT), cloud computing, and artificial intelligence has brought more possibilities of improving and solving the current problems. Data analytics and the use of machine learning/deep learning makes it possible to learn the underlying patterns and make decisions based on what was learned from massive data generated from IoT sensors. Process involves the use of IoT sensors, Raspberry Pis, Amazon Web Services (AWS) and multiple machine learning techniques with the intent to identify anomalous cases for the smart home security system.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid development in the field of System of Chip (SoC) technology, Internet of Things (IoT), cloud computing, and artificial intelligence has brought more possibilities of improving and solving the current problems. With data analytics and the use of machine learning/deep learning, it is made possible to learn the underlying patterns and make decisions based on what was learned from massive data generated from IoT sensors. When combined with cloud computing, the whole pipeline can be automated, and free of manual controls and operations. In this paper, an implementation of an automated data engineering pipeline for anomaly detection of IoT sensor data is studied and proposed. The process involves the use of IoT sensors, Raspberry Pis, Amazon Web Services (AWS) and multiple machine learning techniques with the intent to identify anomalous cases for the smart home security system.

Related papers

• A Plug-in Tiny AI Module for Intelligent and Selective Sensor Data Transmission [10.174575604689391]
  We propose a novel sensing module to equip sensing frameworks with intelligent data transmission capabilities. We integrate a highly efficient machine learning model placed near the sensor. This model provides prompt feedback for the sensing system to transmit only valuable data while discarding irrelevant information.
  arXiv  Detail & Related papers  (2024-02-03T05:41:39Z)
• IoT-based Route Recommendation for an Intelligent Waste Management System [61.04795047897888]
  This work proposes an intelligent approach to route recommendation in an IoT-enabled waste management system given spatial constraints. Our solution is based on a multiple-level decision-making process in which bins' status and coordinates are taken into account.
  arXiv  Detail & Related papers  (2022-01-01T12:36:22Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Automated Machine Learning Techniques for Data Streams [91.3755431537592]
  This paper surveys the state-of-the-art open-source AutoML tools, applies them to data collected from streams, and measures how their performance changes over time. The results show that off-the-shelf AutoML tools can provide satisfactory results but in the presence of concept drift, detection or adaptation techniques have to be applied to maintain the predictive accuracy over time.
  arXiv  Detail & Related papers  (2021-06-14T11:42:46Z)
• Pervasive AI for IoT Applications: Resource-efficient Distributed Artificial Intelligence [45.076180487387575]
  Artificial intelligence (AI) has witnessed a substantial breakthrough in a variety of Internet of Things (IoT) applications and services. This is driven by the easier access to sensory data and the enormous scale of pervasive/ubiquitous devices that generate zettabytes (ZB) of real-time data streams. The confluence of pervasive computing and artificial intelligence, Pervasive AI, expanded the role of ubiquitous IoT systems.
  arXiv  Detail & Related papers  (2021-05-04T23:42:06Z)
• IoT Security: Botnet detection in IoT using Machine learning [0.0]
  This research work is to propose an innovative model using machine learning algorithm to detect and mitigate botnet-based distributed denial of service (DDoS) attack in IoT network. Our proposed model tackles the security issue concerning the threats from bots.
  arXiv  Detail & Related papers  (2021-04-06T01:47:50Z)
• Reliable Fleet Analytics for Edge IoT Solutions [0.0]
  We propose a framework for facilitating machine learning at the edge for AIoT applications. The contribution is an architecture that includes services, tools, and methods for delivering fleet analytics at scale. We present a preliminary validation of the framework by performing experiments with IoT devices on a university campus's rooms.
  arXiv  Detail & Related papers  (2021-01-12T11:28:43Z)
• Machine Learning in the Internet of Things for Industry 4.0 [0.0]
  We show that organization of such systems depends on the entire processing stack, from the hardware layer all the way to the software layer, as well as on the required response times of the IoT system. We propose a flow processing stack for such systems along with the organizational machine learning architectural patterns that enable the possibility to spread the learning and inferencing on the edge and the cloud.
  arXiv  Detail & Related papers  (2020-05-22T12:43:15Z)
• Contextual-Bandit Anomaly Detection for IoT Data in Distributed Hierarchical Edge Computing [65.78881372074983]
  IoT devices can hardly afford complex deep neural networks (DNN) models, and offloading anomaly detection tasks to the cloud incurs long delay. We propose and build a demo for an adaptive anomaly detection approach for distributed hierarchical edge computing (HEC) systems. We show that our proposed approach significantly reduces detection delay without sacrificing accuracy, as compared to offloading detection tasks to the cloud.
  arXiv  Detail & Related papers  (2020-04-15T06:13:33Z)
• Adaptive Anomaly Detection for IoT Data in Hierarchical Edge Computing [71.86955275376604]
  We propose an adaptive anomaly detection approach for hierarchical edge computing (HEC) systems to solve this problem. We design an adaptive scheme to select one of the models based on the contextual information extracted from input data, to perform anomaly detection. We evaluate our proposed approach using a real IoT dataset, and demonstrate that it reduces detection delay by 84% while maintaining almost the same accuracy as compared to offloading detection tasks to the cloud.
  arXiv  Detail & Related papers  (2020-01-10T05:29:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.