Practical validation of synthetic pre-crash scenarios

• URL: http://arxiv.org/abs/2605.04564v1
• Date: Wed, 06 May 2026 07:08:06 GMT
• Title: Practical validation of synthetic pre-crash scenarios
• Abstract summary: A gap remains in the robust evaluation of synthetic pre-crash scenarios' practical equivalence to their real-world counterparts.<n>Conventional significance testing is inadequate, as it focuses on detecting differences rather than establishing practical equivalence.<n>This study introduces a binning-based approach to define appropriate statistics and equivalence criteria.
• Score: 5.031637169030579
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The representativeness of synthetic pre-crash scenarios is crucial for assessing the safety impact of Driving Automation Systems through virtual simulations. However, a gap remains in the robust evaluation of synthetic pre-crash scenarios' practical equivalence to their real-world counterparts; that is, whether they are similar enough for the intended assessment purpose. Conventional significance testing is inadequate, as it focuses on detecting differences rather than establishing practical equivalence. This study addresses the research gap by extending our previous work on a Bayesian Region of Practical Equivalence (ROPE)-based equivalence testing framework by introducing a binning-based approach to define appropriate statistics and equivalence criteria. Two binning-based statistics are proposed to measure practically meaningful distributional differences between datasets in the context of safety impact assessment. The framework's applicability is demonstrated through a case study, which tests the practical equivalence of two synthetic rear-end pre-crash datasets with a previously developed reference dataset in the context of the safety impact assessment of an Automatic Emergency Braking system. The results show that the framework provides informative quantitative assessments of practical equivalence as well as diagnostic insights into the divergence of datasets. Although the demonstration focuses on rear-end pre-crash scenarios, the framework is generic and extensible to broader validation contexts, providing an interpretable and principled basis for practical equivalence assessment across diverse synthetic data applications.

Related papers

• To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias [61.40435736418359]
  We introduce a unified framework that standardizes heterogeneous benchmarks to contrast isolated demographic assessments with forced-choice comparative settings.<n>We show that comparative settings act as aggressive catalysts for latent discrimination, a shift primarily driven by underspecified contexts.<n>Finally, we demonstrate that this comparative prejudice is a generalized phenomenon that scales positively with model size.
  arXiv  Detail & Related papers  (2026-06-23T13:53:50Z)
• Starting Off on the Wrong Foot: Pitfalls in Data Preparation [6.654947289826598]
  This study illustrates that conventional data preparation procedures often yield unreliable and unstable results.<n>We propose a novel data preparation framework leveraging two recent statistical advancements.<n>Our findings demonstrate that incorporating statistically rigorous data preparation methods significantly enhances model robustness and interpretability.
  arXiv  Detail & Related papers  (2026-03-18T18:37:33Z)
• SHAP Distance: An Explainability-Aware Metric for Evaluating the Semantic Fidelity of Synthetic Tabular Data [7.227194225143588]
  We introduce the SHapley Additive exPlanations (SHAP) Distance, a novel explainability-aware metric that is defined as the cosine distance between the global SHAP attribution vectors.<n>We analyze datasets that span clinical health records with physiological features, enterprise invoice transactions with heterogeneous scales, and telecom churn logs with mixed categorical-numerical attributes.<n>Our results show that the SHAP Distance captures feature importance shifts and underrepresented tail effects that the Kullback-Leibler divergence and Train-on-Synthetic-Test-on-Real accuracy fail to detect.
  arXiv  Detail & Related papers  (2025-11-17T03:47:47Z)
• On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles [4.342427756164555]
  This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance.<n>By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions.<n>Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other.
  arXiv  Detail & Related papers  (2025-05-04T22:06:23Z)
• ACTIVA: Amortized Causal Effect Estimation via Transformer-based Variational Autoencoder [7.987204219322316]
  We propose ACTIVA, a conditional variational autoencoder architecture for amortized causal inference.<n>ACTIVA learns a latent representation conditioned on observational inputs and intervention queries, enabling zero-shot inference.<n>We provide theoretical insights showing that ACTIVA predicts interventional distributions as mixtures over observationally equivalent causal models.
  arXiv  Detail & Related papers  (2025-03-03T08:28:25Z)
• Evaluating Inter-Column Logical Relationships in Synthetic Tabular Data Generation [49.898152180805454]
  This paper proposes three evaluation metrics designed to assess the preservation of logical relationships.<n>We validate these metrics by assessing the performance of both classical and state-of-the-art generation methods on a real-world industrial dataset.
  arXiv  Detail & Related papers  (2025-02-06T13:13:26Z)
• Testing Generalizability in Causal Inference [3.547529079746247]
  No formal procedure exists for statistically evaluating generalizability in machine learning algorithms.<n>We propose a systematic framework for statistically evaluating the generalizability of high-dimensional causal inference models.
  arXiv  Detail & Related papers  (2024-11-05T11:44:00Z)
• Hypothesis Testing for Class-Conditional Noise Using Local Maximum Likelihood [1.8798171797988192]
  In supervised learning, automatically assessing the quality of the labels before any learning takes place remains an open research question. In this paper we show how similar procedures can be followed when the underlying model is a product of Local Maximum Likelihood Estimation. This different view allows for wider applicability of the tests by offering users access to a richer model class.
  arXiv  Detail & Related papers  (2023-12-15T22:14:58Z)
• Reliability in Semantic Segmentation: Can We Use Synthetic Data? [69.28268603137546]
  We show for the first time how synthetic data can be specifically generated to assess comprehensively the real-world reliability of semantic segmentation models. This synthetic data is employed to evaluate the robustness of pretrained segmenters. We demonstrate how our approach can be utilized to enhance the calibration and OOD detection capabilities of segmenters.
  arXiv  Detail & Related papers  (2023-12-14T18:56:07Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Systematic Evaluation of Predictive Fairness [60.0947291284978]
  Mitigating bias in training on biased datasets is an important open problem. We examine the performance of various debiasing methods across multiple tasks. We find that data conditions have a strong influence on relative model performance.
  arXiv  Detail & Related papers  (2022-10-17T05:40:13Z)
• Enabling Counterfactual Survival Analysis with Balanced Representations [64.17342727357618]
  Survival data are frequently encountered across diverse medical applications, i.e., drug development, risk profiling, and clinical trials. We propose a theoretically grounded unified framework for counterfactual inference applicable to survival outcomes.
  arXiv  Detail & Related papers  (2020-06-14T01:15:00Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.