Autoencoder-based Online Data Quality Monitoring for the CMS
Electromagnetic Calorimeter
- URL: http://arxiv.org/abs/2308.16659v1
- Date: Thu, 31 Aug 2023 11:58:13 GMT
- Title: Autoencoder-based Online Data Quality Monitoring for the CMS
Electromagnetic Calorimeter
- Authors: Abhirami Harilal, Kyungmin Park, Michael Andrews and Manfred Paulini
(on behalf of the CMS Collaboration)
- Abstract summary: A real-time autoencoder-based anomaly detection system is developed that is able to detect ECAL anomalies unseen in past data.
The new system is able to efficiently detect anomalies while maintaining an estimated false discovery rate between $10-2$ to $10-4$, beating existing benchmarks by about two orders of magnitude.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The online Data Quality Monitoring system (DQM) of the CMS electromagnetic
calorimeter (ECAL) is a crucial operational tool that allows ECAL experts to
quickly identify, localize, and diagnose a broad range of detector issues that
would otherwise hinder physics-quality data taking. Although the existing ECAL
DQM system has been continuously updated to respond to new problems, it remains
one step behind newer and unforeseen issues. Using unsupervised deep learning,
a real-time autoencoder-based anomaly detection system is developed that is
able to detect ECAL anomalies unseen in past data. After accounting for spatial
variations in the response of the ECAL and the temporal evolution of anomalies,
the new system is able to efficiently detect anomalies while maintaining an
estimated false discovery rate between $10^{-2}$ to $10^{-4}$, beating existing
benchmarks by about two orders of magnitude. The real-world performance of the
system is validated using anomalies found in 2018 and 2022 LHC collision data.
Additionally, first results from deploying the autoencoder-based system in the
CMS online DQM workflow for the ECAL barrel during Run 3 of the LHC are
presented, showing its promising performance in detecting obscure issues that
could have been missed in the existing DQM system.
Related papers
- MELODY: Robust Semi-Supervised Hybrid Model for Entity-Level Online Anomaly Detection with Multivariate Time Series [11.754433499581879]
A faulty code change may degrade the target service's performance and cause cascading outages in downstream services.
In this paper, we study the problem of anomaly detection for deployments.
We propose a novel framework, semi-supervised hybrid Model for Entity-Level Online Detection of anomalY (MELODY)
arXiv Detail & Related papers (2024-01-18T19:02:41Z) - Spatio-Temporal Anomaly Detection with Graph Networks for Data Quality
Monitoring of the Hadron Calorimeter [1.1420116121141344]
The CMS experiment is a general-purpose detector for high-energy collision at the large hadron collider (HCL) at CERN.
It employs an online data quality monitoring (DQM) system to promptly spot and diagnose particle data acquisition problems to avoid data quality loss.
We propose the GraphSTAD system, which employs convolutional and graph neural networks to learn local spatial characteristics induced by particles traversing the detector.
arXiv Detail & Related papers (2023-11-07T18:33:08Z) - Autoencoder-based Anomaly Detection System for Online Data Quality Monitoring of the CMS Electromagnetic Calorimeter [0.0]
A real-time autoencoder-based anomaly detection system using semi-supervised machine learning is presented.
A novel method is introduced which maximizes the anomaly detection performance by exploiting the time-dependent evolution of anomalies.
The performance of the system is validated with anomalies found in 2018 and 2022 LHC collision data.
arXiv Detail & Related papers (2023-09-18T21:11:25Z) - A Robust and Explainable Data-Driven Anomaly Detection Approach For
Power Electronics [56.86150790999639]
We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer.
The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data.
A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
arXiv Detail & Related papers (2022-09-23T06:09:35Z) - Measurement based estimator scheme for continuous quantum error
correction [52.77024349608834]
Canonical discrete quantum error correction (DQEC) schemes use projective von Neumann measurements on stabilizers to discretize the error syndromes into a finite set.
Quantum error correction (QEC) based on continuous measurement, known as continuous quantum error correction (CQEC), can be executed faster than DQEC and can also be resource efficient.
We show that by constructing a measurement-based estimator (MBE) of the logical qubit to be protected, it is possible to accurately track the errors occurring on the physical qubits in real time.
arXiv Detail & Related papers (2022-03-25T09:07:18Z) - Smart Metering System Capable of Anomaly Detection by Bi-directional
LSTM Autoencoder [0.6649973446180738]
Anomaly detection is concerned with a wide range of applications such as fault detection, system monitoring, and event detection.
This paper presents an anomaly detection process to find outliers observed in the smart metering system.
BiLSTM based autoencoder is used and finds the anomalous data point.
arXiv Detail & Related papers (2021-12-06T12:34:59Z) - Cloud Failure Prediction with Hierarchical Temporary Memory: An
Empirical Assessment [64.73243241568555]
Hierarchical Temporary Memory (HTM) is an unsupervised learning algorithm inspired by the features of the neocortex.
This paper presents the first systematic study that assesses HTM in the context of failure prediction.
arXiv Detail & Related papers (2021-10-06T07:09:45Z) - DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly
detection in air transportation [68.8204255655161]
We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE)
It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase.
Results show that the DAE achieves better results in both accuracy and speed of detection.
arXiv Detail & Related papers (2021-09-08T14:07:55Z) - TELESTO: A Graph Neural Network Model for Anomaly Classification in
Cloud Services [77.454688257702]
Machine learning (ML) and artificial intelligence (AI) are applied on IT system operation and maintenance.
One direction aims at the recognition of re-occurring anomaly types to enable remediation automation.
We propose a method that is invariant to dimensionality changes of given data.
arXiv Detail & Related papers (2021-02-25T14:24:49Z) - Anomaly Detection using Deep Autoencoders for in-situ Wastewater Systems
Monitoring Data [0.0]
This paper proposes an anomaly detection method based on a deep autoencoder for in-situ wastewater systems monitoring data.
Anomaly detection is then performed based on the reconstruction error of the decoding stage.
arXiv Detail & Related papers (2020-02-07T09:53:46Z) - Adaptive Anomaly Detection for IoT Data in Hierarchical Edge Computing [71.86955275376604]
We propose an adaptive anomaly detection approach for hierarchical edge computing (HEC) systems to solve this problem.
We design an adaptive scheme to select one of the models based on the contextual information extracted from input data, to perform anomaly detection.
We evaluate our proposed approach using a real IoT dataset, and demonstrate that it reduces detection delay by 84% while maintaining almost the same accuracy as compared to offloading detection tasks to the cloud.
arXiv Detail & Related papers (2020-01-10T05:29:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.