Robustness and Generalization Performance of Deep Learning Models on
Cyber-Physical Systems: A Comparative Study
- URL: http://arxiv.org/abs/2306.07737v1
- Date: Tue, 13 Jun 2023 12:43:59 GMT
- Title: Robustness and Generalization Performance of Deep Learning Models on
Cyber-Physical Systems: A Comparative Study
- Authors: Alexander Windmann and Henrik Steude and Oliver Niggemann
- Abstract summary: Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise.
We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
- Score: 71.84852429039881
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep learning (DL) models have seen increased attention for time series
forecasting, yet the application on cyber-physical systems (CPS) is hindered by
the lacking robustness of these methods. Thus, this study evaluates the
robustness and generalization performance of DL architectures on multivariate
time series data from CPS. Our investigation focuses on the models' ability to
handle a range of perturbations, such as sensor faults and noise, and assesses
their impact on overall performance. Furthermore, we test the generalization
and transfer learning capabilities of these models by exposing them to
out-of-distribution (OOD) samples. These include deviations from standard
system operations, while the core dynamics of the underlying physical system
are preserved. Additionally, we test how well the models respond to several
data augmentation techniques, including added noise and time warping. Our
experimental framework utilizes a simulated three-tank system, proposed as a
novel benchmark for evaluating the robustness and generalization performance of
DL algorithms in CPS data contexts. The findings reveal that certain DL model
architectures and training techniques exhibit superior effectiveness in
handling OOD samples and various perturbations. These insights have significant
implications for the development of DL models that deliver reliable and robust
performance in real-world CPS applications.
Related papers
- Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
We propose a novel approach to explain the behavior of a black-box model under feature shifts.
We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
arXiv Detail & Related papers (2024-08-24T18:28:19Z) - Impacts of Data Preprocessing and Hyperparameter Optimization on the Performance of Machine Learning Models Applied to Intrusion Detection Systems [0.8388591755871736]
Intrusion Detection Systems (IDS) have been continuously improved.
Many of them incorporate machine learning (ML) techniques to identify threats.
This article aims to present a study that fills this research gap.
arXiv Detail & Related papers (2024-07-15T14:30:25Z) - Impact of Architectural Modifications on Deep Learning Adversarial Robustness [16.991522358940774]
We present an experimental evaluation of the effects of model modifications on deep learning model robustness using adversarial attacks.
Our results indicate the pressing demand for an in-depth assessment of the effects of model changes on the robustness of models.
arXiv Detail & Related papers (2024-05-03T08:58:38Z) - Enhancing Dynamical System Modeling through Interpretable Machine
Learning Augmentations: A Case Study in Cathodic Electrophoretic Deposition [0.8796261172196743]
We introduce a comprehensive data-driven framework aimed at enhancing the modeling of physical systems.
As a demonstrative application, we pursue the modeling of cathodic electrophoretic deposition (EPD), commonly known as e-coating.
arXiv Detail & Related papers (2024-01-16T14:58:21Z) - A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime.
We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
arXiv Detail & Related papers (2023-11-13T01:48:08Z) - CoDBench: A Critical Evaluation of Data-driven Models for Continuous
Dynamical Systems [8.410938527671341]
We introduce CodBench, an exhaustive benchmarking suite comprising 11 state-of-the-art data-driven models for solving differential equations.
Specifically, we evaluate 4 distinct categories of models, viz., feed forward neural networks, deep operator regression models, frequency-based neural operators, and transformer architectures.
We conduct extensive experiments, assessing the operators' capabilities in learning, zero-shot super-resolution, data efficiency, robustness to noise, and computational efficiency.
arXiv Detail & Related papers (2023-10-02T21:27:54Z) - Reinforcement Learning for Topic Models [3.42658286826597]
We apply reinforcement learning techniques to topic modeling by replacing the variational autoencoder in ProdLDA with a continuous action space reinforcement learning policy.
We introduce several modifications: modernize the neural network architecture, weight the ELBO loss, use contextual embeddings, and monitor the learning process via computing topic diversity and coherence.
arXiv Detail & Related papers (2023-05-08T16:41:08Z) - SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in
Fine-tuned Source Code Models [58.78043959556283]
We study the behaviors of models under different fine-tuning methodologies, including full fine-tuning and Low-Rank Adaptation (LoRA) fine-tuning methods.
Our analysis uncovers that LoRA fine-tuning consistently exhibits significantly better OOD generalization performance than full fine-tuning across various scenarios.
arXiv Detail & Related papers (2022-10-10T16:07:24Z) - How robust are pre-trained models to distribution shift? [82.08946007821184]
We show how spurious correlations affect the performance of popular self-supervised learning (SSL) and auto-encoder based models (AE)
We develop a novel evaluation scheme with the linear head trained on out-of-distribution (OOD) data, to isolate the performance of the pre-trained models from a potential bias of the linear head used for evaluation.
arXiv Detail & Related papers (2022-06-17T16:18:28Z) - Leveraging the structure of dynamical systems for data-driven modeling [111.45324708884813]
We consider the impact of the training set and its structure on the quality of the long-term prediction.
We show how an informed design of the training set, based on invariants of the system and the structure of the underlying attractor, significantly improves the resulting models.
arXiv Detail & Related papers (2021-12-15T20:09:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.