One Permutation Is All You Need: Fast, Reliable Variable Importance and Model Stress-Testing

• URL: http://arxiv.org/abs/2512.13892v2
• Date: Tue, 23 Dec 2025 12:54:15 GMT
• Title: One Permutation Is All You Need: Fast, Reliable Variable Importance and Model Stress-Testing
• Authors: Albert Dorador,
• Abstract summary: We show that by replacing multiple random permutations with a single, deterministic and optimal permutation, we achieve a method that retains the core principles of permutation-based importance.<n>We validate this approach across nearly 200 scenarios, including real-world household finance and credit risk applications.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reliable estimation of feature contributions in machine learning models is essential for trust, transparency and regulatory compliance, especially when models are proprietary or otherwise operate as black boxes. While permutation-based methods are a standard tool for this task, classical implementations rely on repeated random permutations, introducing computational overhead and stochastic instability. In this paper, we show that by replacing multiple random permutations with a single, deterministic, and optimal permutation, we achieve a method that retains the core principles of permutation-based importance while being non-random, faster, and more stable. We validate this approach across nearly 200 scenarios, including real-world household finance and credit risk applications, demonstrating improved bias-variance tradeoffs and accuracy in challenging regimes such as small sample sizes, high dimensionality, and low signal-to-noise ratios. Finally, we introduce Systemic Variable Importance, a natural extension designed for model stress-testing that explicitly accounts for feature correlations. This framework provides a transparent way to quantify how shocks or perturbations propagate through correlated inputs, revealing dependencies that standard variable importance measures miss. Two real-world case studies demonstrate how this metric can be used to audit models for hidden reliance on protected attributes (e.g., gender or race), enabling regulators and practitioners to assess fairness and systemic risk in a principled and computationally efficient manner.

Related papers

• The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI [0.0]
  This paper introduces a novel auditing framework to quantify latent trait estimation under ordinal uncertainty.<n>The research audits nine leading models across dimensions including Optimization Bias, Sycophancy, and Status-Quo Legitimization.
  arXiv  Detail & Related papers  (2026-02-19T06:56:01Z)
• Revisiting Randomization in Greedy Model Search [16.15551706774035]
  We propose and analyze an ensemble of greedy forward selection estimators that are randomized by feature subsampling.<n>We design a novel implementation based on dynamic programming that greatly improves its computational efficiency.<n>Contrary to prevailing belief that randomized ensembling is analogous to shrinkage, we show that it can simultaneously reduce training error and degrees of freedom.
  arXiv  Detail & Related papers  (2025-06-18T17:13:53Z)
• Network Inversion for Generating Confidently Classified Counterfeits [11.599035626374409]
  In vision classification, generating inputs that elicit confident predictions is key to understanding model behavior and reliability.<n>We extend network inversion techniques to generate Confidently Classified Counterfeits (CCCs)<n>CCCs offer a model-centric perspective on confidence, revealing that models can assign high confidence to entirely synthetic, out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-03-26T03:26:49Z)
• Quantifying Uncertainty and Variability in Machine Learning: Confidence Intervals for Quantiles in Performance Metric Distributions [0.17265013728931003]
  Machine learning models are widely used in applications where reliability and robustness are critical.<n>Model evaluation often relies on single-point estimates of performance metrics that fail to capture the inherent variability in model performance.<n>This contribution explores the use of quantiles and confidence intervals to analyze such distributions, providing a more complete understanding of model performance and its uncertainty.
  arXiv  Detail & Related papers  (2025-01-28T13:21:34Z)
• Regularization for Adversarial Robust Learning [18.46110328123008]
  We develop a novel approach to adversarial training that integrates $phi$-divergence regularization into the distributionally robust risk function. This regularization brings a notable improvement in computation compared with the original formulation. We validate our proposed method in supervised learning, reinforcement learning, and contextual learning and showcase its state-of-the-art performance against various adversarial attacks.
  arXiv  Detail & Related papers  (2024-08-19T03:15:41Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Robustness of Machine Learning Models Beyond Adversarial Attacks [0.0]
  We show that the widely used concept of adversarial robustness and closely related metrics are not necessarily valid metrics for determining the robustness of ML models. We propose a flexible approach that models possible perturbations in input data individually for each application. This is then combined with a probabilistic approach that computes the likelihood that a real-world perturbation will change a prediction.
  arXiv  Detail & Related papers  (2022-04-21T12:09:49Z)
• Probabilistic robust linear quadratic regulators with Gaussian processes [73.0364959221845]
  Probabilistic models such as Gaussian processes (GPs) are powerful tools to learn unknown dynamical systems from data for subsequent use in control design. We present a novel controller synthesis for linearized GP dynamics that yields robust controllers with respect to a probabilistic stability margin.
  arXiv  Detail & Related papers  (2021-05-17T08:36:18Z)
• Causally-motivated Shortcut Removal Using Auxiliary Labels [63.686580185674195]
  Key challenge to learning such risk-invariant predictors is shortcut learning. We propose a flexible, causally-motivated approach to address this challenge. We show both theoretically and empirically that this causally-motivated regularization scheme yields robust predictors.
  arXiv  Detail & Related papers  (2021-05-13T16:58:45Z)
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning [123.3472310767721]
  Prediction credibility measures are fundamental in statistics and machine learning. These measures should account for the wide variety of models used in practice. The framework developed in this work expresses the credibility as a risk-fit trade-off.
  arXiv  Detail & Related papers  (2020-11-24T19:52:38Z)
• Hidden Cost of Randomized Smoothing [72.93630656906599]
  In this paper, we point out the side effects of current randomized smoothing. Specifically, we articulate and prove two major points: 1) the decision boundaries of smoothed classifiers will shrink, resulting in disparity in class-wise accuracy; 2) applying noise augmentation in the training process does not necessarily resolve the shrinking issue due to the inconsistent learning objectives.
  arXiv  Detail & Related papers  (2020-03-02T23:37:42Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.