Synthetic Designed Experiments for Diagnosing Vision Model Failure

• URL: http://arxiv.org/abs/2605.00832v1
• Date: Mon, 30 Mar 2026 02:52:21 GMT
• Title: Synthetic Designed Experiments for Diagnosing Vision Model Failure
• Abstract summary: Current synthetic data pipelines for computer vision generate images without diagnosing what the downstream model actually needs.<n>This open-loop paradigm treats synthetic data as cheap real data, randomly sampling the generator's output space.<n>We argue this fundamentally misuses synthetic data's unique property: the controllable, independent variation of scene factors.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current synthetic data pipelines for computer vision generate images without diagnosing what the downstream model actually needs. This open-loop paradigm treats synthetic data as cheap real data, randomly sampling the generator's output space and hoping to cover the model's failure modes. We argue this fundamentally misuses synthetic data's unique property: the controllable, independent variation of scene factors.Drawing on the statistical theory of Design of Experiments (DoE), we propose Synthetic Designed Experiments for Representational Sufficiency (SDRS). SDRS treats the downstream model as a black-box system and the synthetic generator as an experimental apparatus. Using fractional factorial designs, SDRS efficiently audits a model's factor-sensitivity profile via ANOVA decomposition. It classifies failures into two actionable types: Type I gaps (coverage failures on underrepresented factor levels) and Type II gaps (reliance on spurious nuisance dependencies). The audit then prescribes targeted synthetic data to address each gap type. We validate SDRS on three experiments: (1) a controlled diagnostic on dSprites with planted biases, where the audit correctly identifies both gap types and targeted data improves accuracy from 49.9% to 79.0%; (2) a dense segmentation task on procedural scenes, where detecting background-complexity shortcuts and applying targeted data improves mIoU from 0.948 to 0.998; and (3) an entanglement detection experiment showing that the ANOVA audit identifies cross-factor contamination in imperfect generators. Finally, we show that per-factor invariance penalties can transfer sensitivity between factors, identifying an open problem for representation-level correction.

Related papers

• Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data [73.19814469893088]
  Digital twins and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems.<n>This paper addresses whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible.<n>Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing.
  arXiv  Detail & Related papers  (2026-08-28T07:05:38Z)
• Function-Space Decoupled Diffusion for Forward and Inverse Modeling in Carbon Capture and Storage [65.51149575007149]
  We present Fun-DDPS, a generative framework that combines function-space diffusion models with differentiable neural operator surrogates for both forward and inverse modeling.<n>Fun-DDPS produces physically consistent realizations free from the high-frequency artifacts observed in joint-state baselines.
  arXiv  Detail & Related papers  (2026-02-12T18:58:12Z)
• Crucial-Diff: A Unified Diffusion Model for Crucial Image and Annotation Synthesis in Data-scarce Scenarios [65.97836905826145]
  scarcity of data in various scenarios, such as medical, industry and autonomous driving, leads to model overfitting and dataset imbalance.<n>We propose Crucial-Diff, a domain-agnostic framework designed to synthesize crucial samples.<n>Our framework generates diverse, high-quality training data, achieving a pixel-level AP of 83.63% and an F1-MAX of 78.12% on MVTec.
  arXiv  Detail & Related papers  (2025-07-14T04:41:38Z)
• Generating on Generated: An Approach Towards Self-Evolving Diffusion Models [58.05857658085845]
  Recursive Self-Improvement (RSI) enables intelligence systems to autonomously refine their capabilities.<n>This paper explores the application of RSI in text-to-image diffusion models, addressing the challenge of training collapse caused by synthetic data.
  arXiv  Detail & Related papers  (2025-02-14T07:41:47Z)
• Leveraging Latent Diffusion Models for Training-Free In-Distribution Data Augmentation for Surface Defect Detection [9.784793380119806]
  We introduce DIAG, a training-free Diffusion-based In-distribution Anomaly Generation pipeline for data augmentation. Unlike conventional image generation techniques, we implement a human-in-the-loop pipeline, where domain experts provide multimodal guidance to the model. We demonstrate the efficacy and versatility of DIAG with respect to state-of-the-art data augmentation approaches on the challenging KSDD2 dataset.
  arXiv  Detail & Related papers  (2024-07-04T14:28:52Z)
• Diffusion-based Image Generation for In-distribution Data Augmentation in Surface Defect Detection [8.93281936150572]
  We show that diffusion models can be used in industrial scenarios to improve the data augmentation procedure. We propose a novel approach for data augmentation that mixes out-of-distribution with in-distribution samples.
  arXiv  Detail & Related papers  (2024-06-01T17:09:18Z)
• Fault Detection and Monitoring using a Data-Driven Information-Based Strategy: Method, Theory, and Application [5.056456697289351]
  We propose an information-driven fault detection method based on a novel concept drift detector.<n>The method is tailored to identifying drifts in input-output relationships of additive noise models.<n>We prove several theoretical properties of the proposed MI-based fault detection scheme.
  arXiv  Detail & Related papers  (2024-05-06T17:43:39Z)
• A Comparison of Residual-based Methods on Fault Detection [6.675805308519987]
  In this study, we compare two residual-based approaches to detect faults in industrial systems. The performance evaluation focuses on three tasks: health indicator construction, fault detection, and health indicator interpretation. The detection results reveal that both models are capable of detecting faults with an average delay of around 20 cycles and maintain a low false positive rate.
  arXiv  Detail & Related papers  (2023-09-05T14:39:27Z)
• Breaking the Spurious Causality of Conditional Generation via Fairness Intervention with Corrective Sampling [77.15766509677348]
  Conditional generative models often inherit spurious correlations from the training dataset. This can result in label-conditional distributions that are imbalanced with respect to another latent attribute. We propose a general two-step strategy to mitigate this issue.
  arXiv  Detail & Related papers  (2022-12-05T08:09:33Z)
• Fault Detection and Diagnosis with Imbalanced and Noisy Data: A Hybrid Framework for Rotating Machinery [2.580765958706854]
  Fault diagnosis plays an essential role in reducing the maintenance costs of rotating machinery manufacturing systems. Traditional Fault Detection and Diagnosis (FDD) frameworks get poor performances when dealing with real-world circumstances. This paper proposes a hybrid framework which uses the three aforementioned components to achieve an effective signal-based FDD system.
  arXiv  Detail & Related papers  (2022-02-09T01:09:59Z)
• SUOD: Accelerating Large-Scale Unsupervised Heterogeneous Outlier Detection [63.253850875265115]
  Outlier detection (OD) is a key machine learning (ML) task for identifying abnormal objects from general samples. We propose a modular acceleration system, called SUOD, to address it.
  arXiv  Detail & Related papers  (2020-03-11T00:22:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.