From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol

• URL: http://arxiv.org/abs/2606.24996v1
• Date: Tue, 23 Jun 2026 15:59:17 GMT
• Title: From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol
• Abstract summary: We study when a forecast-side winner can be certified as deployment-actionable for a specified interface and deployed utility.<n>Traffic-Hourly provides a certified anchor: winners agree at zero friction, but positive switching friction makes the forecast winner deployed-suboptimal.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Forecasting leaderboards rank models by predictive quality, but their winners are often read as deployment-ready top-1 advice. That reading can fail when forecasts are passed through a fixed decision interface, such as an alert threshold, a top-k budget, or a switching-cost policy. We study when a forecast-side winner can be certified as deployment-actionable for a specified interface and deployed utility. We introduce a fail-closed certification protocol whose gates are sufficient evidential conditions for a strong claim: a friction-caused, non-tie, statistically supported, and recurrent deployment-side reversal. Traffic-Hourly provides a certified anchor: winners agree at zero friction, but positive switching friction makes the forecast winner deployed-suboptimal. A locked native audit tests overclaiming: across 22 verified candidates and 362 full-grid cells, 155 apparent forecast/deployment winner inversions are blocked before certification. The contribution is not a new forecaster, metric, or universal utility, but a conservative protocol for deciding when forecasting leaderboard winners should be read as deployment-actionable top-1 advice.

Related papers

• Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection [0.0]
  Promotion is security-relevant because it changes the model responsible for subsequent attack detection.<n>We test dependence on CICIDS 2017, UNSW-NB15 and ToN-IoT with self-contained challenger pipelines.<n>Raising nominal candidate evidence from 512 to 2,000 samples per class improved promotion under pool-constructed progressive drift by +0.53, +1.67 and +0.38.
  arXiv  Detail & Related papers  (2026-09-03T18:55:09Z)
• When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design [8.037395295172088]
  Design campaigns in chemistry, materials science, and machine learning share a bottleneck: determining how good a candidate truly is requires an expensive evaluation.<n>Machine-learning surrogates that predict these outcomes are increasingly used to propose candidates but to grade them, and even to feed their own predictions back into the search as though they were measurements.<n>We establish when this practice is safe, what any certificate of safety must cost, and when the substitution provably pays.
  arXiv  Detail & Related papers  (2026-08-02T16:59:29Z)
• RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses [7.123785374544969]
  Large language models (LLMs) make reward design in reinforcement learning substantially more scalable, but generated rewards are not automatically reliable training objectives.<n>We study this deployment-time problem by treating generated rewards as reward hypotheses whose utility depends on the competence of the current policy.<n>We propose textscRHyVE, a competence-aware verification and phase-aware deployment protocol.
  arXiv  Detail & Related papers  (2026-04-30T16:01:51Z)
• Scalable and Precise Patch Robustness Certification for Deep Learning Models with Top-k Predictions [2.6499018693213316]
  Patch robustness certification is an emerging verification approach for defending against adversarial patch attacks.<n>We propose CostCert, a voting-based certified recovery defender.<n>We show that CostCert significantly outperforms the current state-of-the-art defender PatchGuard.
  arXiv  Detail & Related papers  (2025-07-31T08:31:59Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Optimal Conformal Prediction under Epistemic Uncertainty [61.46247583794497]
  Conformal prediction (CP) is a popular framework for representing uncertainty.<n>We introduce Bernoulli prediction sets (BPS) which produce the smallest prediction sets that ensure conditional coverage.<n>When given first-order predictions, BPS reduces to the well-known adaptive prediction sets (APS)
  arXiv  Detail & Related papers  (2025-05-25T08:32:44Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably.<n>This poses a significant challenge to ensuring their safe deployment.<n>We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions.<n>We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration [107.15813002403905]
  Decision makers often need to rely on imperfect probabilistic forecasts. We propose a compensation mechanism ensuring that the forecasted utility matches the actually accrued utility. We demonstrate an application showing how passengers could confidently optimize individual travel plans based on flight delay probabilities.
  arXiv  Detail & Related papers  (2020-11-15T08:22:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.