Probabilistic data quality assessment for structural monitoring data via outlier-resistant conditional diffusion model

• URL: http://arxiv.org/abs/2604.26366v1
• Date: Wed, 29 Apr 2026 07:26:37 GMT
• Title: Probabilistic data quality assessment for structural monitoring data via outlier-resistant conditional diffusion model
• Abstract summary: This study proposes a prediction deviation-based SHM data quality assessment method.<n>The proposed conditional diffusion model (CDM) augments the standard diffusion model to incorporate temporal context.<n>The framework significantly improves the accuracy of data quality assessment.
• Score: 9.184776098919471
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Data quality assessment is an essential step that ensures the reliability of the subsequent structural health monitoring (SHM) tasks. This study proposes a prediction deviation-based SHM data quality assessment method using a univariate implicit auto-regressive model, enabling outlier diagnosis and data cleaning. The proposed conditional diffusion model (CDM) augments the standard diffusion model with a conditional embedding module to incorporate temporal context, quartile normalization to mitigate distribution skew, and a Huber loss to enhance robustness against outliers. Within this univariate implicit autoregressive framework, each data point is assigned an outlier probability, quantifying its degree of "outlier-ness", and a global quality evaluation score is computed to characterize the overall dataset quality. Extensive case studies utilizing operational data from real-world structures demonstrate that the proposed framework significantly improves the accuracy of data quality assessment, outperforming other strong baselines representative of clustering, isolation-based, and deep reconstruction methods. The effectiveness and robustness of the proposed framework are further demonstrated by the findings of ablation experiments and hyperparameter analysis.

Related papers

• The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models [0.1274452325287335]
  We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines.<n>PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns.<n> NR-Kfold analysis reveals systematic accuracy loss and increased variability when training-test similarity is broken.
  arXiv  Detail & Related papers  (2026-07-05T16:49:09Z)
• TimeLAVA: Learning-Agnostic Data Valuation for Time Series [48.76711740818533]
  TimeLAVA is a learning-agnostic framework that values temporal segments by their marginal contribution to minimize distributional discrepancy between evaluated and reference data.<n>We provide theoretical guarantees linking valuation to model-agnostic generalization and prove bounded sensitivity to contamination.
  arXiv  Detail & Related papers  (2026-06-17T06:16:16Z)
• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• Heterogeneous Multisource Transfer Learning via Model Averaging for Positive-Unlabeled Data [2.030810815519794]
  We propose a novel transfer learning framework that integrates information from heterogeneous data sources without direct data sharing.<n>For each source domain type, a tailored logistic regression model is conducted, and knowledge is transferred to the PU target domain through model averaging.<n>Our method outperforms other comparative methods in terms of predictive accuracy and robustness, especially under limited labeled data and heterogeneous environments.
  arXiv  Detail & Related papers  (2025-11-14T03:15:31Z)
• Improving Generative Methods for Causal Evaluation via Simulation-Based Inference [9.22158780078584]
  Existing generative methods produce synthetic datasets anchored in observed data (source data)<n>We introduce simulation-based inference for causal evaluation (SBICE), a framework that models generative parameters as uncertain and infers their posterior distribution given a source dataset.<n>SBICE improves the reliability of estimator evaluations by generating more realistic datasets, which supports a robust and data-consistent approach to causal benchmarking under uncertainty.
  arXiv  Detail & Related papers  (2025-09-02T23:35:22Z)
• How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence [23.019102917957152]
  Kernel Divergence Score (KDS) is a novel method that evaluates dataset contamination by computing the divergence between the kernel similarity matrix of sample embeddings.<n>KDS demonstrates a near-perfect correlation with contamination levels and outperforms existing baselines.
  arXiv  Detail & Related papers  (2025-02-02T05:50:39Z)
• Testing and Improving the Robustness of Amortized Bayesian Inference for Cognitive Models [0.5223954072121659]
  Contaminant observations and outliers often cause problems when estimating the parameters of cognitive models.<n>In this study, we test and improve the robustness of parameter estimation using amortized Bayesian inference.<n>The proposed method is straightforward and practical to implement and has a broad applicability in fields where outlier detection or removal is challenging.
  arXiv  Detail & Related papers  (2024-12-29T21:22:24Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Decomposed Adversarial Learned Inference [118.27187231452852]
  We propose a novel approach, Decomposed Adversarial Learned Inference (DALI) DALI explicitly matches prior and conditional distributions in both data and code spaces. We validate the effectiveness of DALI on the MNIST, CIFAR-10, and CelebA datasets.
  arXiv  Detail & Related papers  (2020-04-21T20:00:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.