Reliable Remediation Impact Prediction for Black-Box Security Ratings

• URL: http://arxiv.org/abs/2607.16357v1
• Date: Fri, 17 Jul 2026 09:58:38 GMT
• Title: Reliable Remediation Impact Prediction for Black-Box Security Ratings
• Abstract summary: We propose a surrogate based approach for remediation score impact prediction.<n>We evaluate the approach on a real-world dataset of 5,188 organization configurations from a commercial security rating platform.
• Score: 0.3044748118797888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Security rating platforms summarize externally observable cyber exposure and are expected to help organizations prioritize remediation. A platform may want to tell an organization how a candidate remediation action would affect its score, but repeatedly exposing exact score responses can reveal information about the hidden scoring engine. We propose a surrogate based approach for remediation score impact prediction that is designed to respect this opacity constraint. The surrogate predicts scores from organization configurations while explicitly representing checkpoint (i.e., a security check) applicability and the observed checkpoint set. A main challenge is that such predictions are not uniformly reliable: they depend on the amount and structure of the observable checkpoint evidence available for a given configuration. To address this, the approach combines applicability-aware surrogate construction, sensitivity analysis under controlled checkpoint restriction, a reliability layer for identifying unstable predictions, and score-impact prediction for supported remediation actions. Explicit modeling of checkpoint applicability is central throughout: it improves score prediction and provides the feature basis used by the reliability layer to identify unstable cases. We evaluate the approach on a real-world dataset of 5,188 organization configurations from a commercial security rating platform. The results show that the applicability-aware surrogate improves score prediction over simpler feature representations. For remediation, the surrogate predicts the score impact of supported actions, while the reliability layer helps identify cases in which these predicted impacts should be interpreted cautiously.

Related papers

• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Conformal Reliability: A New Evaluation Metric for Conditional Generation [62.761166941396844]
  Conditional generative models have recently remarkable success in various applications.<n>We propose a novel evaluation metric called reliability based on prediction set at a pre-specified confidence level.<n>We introduce Conformal ReLiability (CReL), a framework that can (i) construct a prediction set and (ii) accurately optimize the reliability score within the constructed prediction set.
  arXiv  Detail & Related papers  (2026-05-29T03:52:44Z)
• Interval-Based AUC (iAUC): Extending ROC Analysis to Uncertainty-Aware Classification [12.024101882027466]
  We propose an uncertainty-aware ROC framework specifically for interval-valued predictions.<n>We introduce two new measures: $AUC_L$ and $AUC_U$.<n>We prove that under valid class-conditional coverage, $AUC_L$ and $AUC_U$ provide formal lower and upper bounds on the theoretical optimal AUC.
  arXiv  Detail & Related papers  (2026-02-04T17:12:04Z)
• Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning [1.2183405753834562]
  This thesis investigates how uncertainty estimation can enhance the safety and trustworthiness of machine learning (ML) systems.<n>We first show that a model's training trajectory contains rich uncertainty signals that can be exploited without altering its architecture or loss.<n>We propose a lightweight, post-hoc abstention method that works across tasks, avoids the cost of deep ensembles, and achieves state-of-the-art selective prediction performance.
  arXiv  Detail & Related papers  (2025-08-11T02:33:53Z)
• Statistical Inference for Responsiveness Verification [15.571656327462142]
  We introduce a formal validation procedure for the responsiveness of predictions with respect to interventions on their features.<n>We describe how to estimate responsiveness for the predictions of any model and any dataset using only black-box access.<n>We develop algorithms that construct these estimates by generating a uniform sample of reachable points.
  arXiv  Detail & Related papers  (2025-07-02T21:50:08Z)
• Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention [65.47632669243657]
  A dishonest institution can exploit mechanisms to discriminate or unjustly deny services under the guise of uncertainty.<n>We demonstrate the practicality of this threat by introducing an uncertainty-inducing attack called Mirage.<n>We propose Confidential Guardian, a framework that analyzes calibration metrics on a reference dataset to detect artificially suppressed confidence.
  arXiv  Detail & Related papers  (2025-05-29T19:47:50Z)
• Aurora: Are Android Malware Classifiers Reliable and Stable under Distribution Shift? [51.12297424766236]
  AURORA is a framework to evaluate malware classifiers based on their confidence quality and operational resilience.<n>AURORA is complemented by a set of metrics designed to go beyond point-in-time performance.<n>The fragility in SOTA frameworks across datasets of varying drift suggests the need for a return to the whiteboard.
  arXiv  Detail & Related papers  (2025-05-28T20:22:43Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably.<n>This poses a significant challenge to ensuring their safe deployment.<n>We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Per-frame mAP Prediction for Continuous Performance Monitoring of Object Detection During Deployment [6.166295570030645]
  We propose an introspection approach to performance monitoring during deployment. We do so by predicting when the per-frame mean average precision drops below a critical threshold. We quantitatively evaluate and demonstrate our method's ability to reduce risk by trading off making an incorrect decision.
  arXiv  Detail & Related papers  (2020-09-18T06:37:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.