NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims

• URL: http://arxiv.org/abs/2605.08192v1
• Date: Tue, 05 May 2026 17:25:28 GMT
• Title: NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims
• Abstract summary: Most consequential claims in AI safety are often the least reproducible.<n>NeurIPS should require standards for papers making such claims.<n>We propose a three-tier disclosure framework, distinguishing public, controlled, and claim-restricted disclosure.
• Score: 11.663456969895462
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier AI safety claims - published assertions that a highly capable general-purpose model is below a threshold of concern, adequately mitigated, or suitable for release - increasingly shape model deployment, governance, and public trust. Yet the artefacts needed to evaluate them are routinely withheld, producing an evidential inversion: the most consequential claims in AI safety are often the least reproducible. This position paper argues that NeurIPS should require reproducibility standards for papers making such claims, treating non-reproducibility not as a transparency preference but as an evaluation-methodology failure. The 2026 International AI Safety Report [Bengio et al., 2026] concludes that reliable pre-deployment safety testing has become harder to conduct and that models now distinguish test from deployment contexts; the 2025 Foundation Model Transparency Index [Wan et al., 2025] reports a sector-average transparency score of 40/100 with no major developer adequately disclosing train-test overlap; contemporaneous measurement-theory work shows that attack-success-rate comparisons across systems are often founded on low-validity measurements [Chouldechova et al., 2025]. We propose a three-tier disclosure framework, distinguishing public, controlled, and claim-restricted disclosure, paired with a mandatory claim inventory, scope statements, and a phased implementation path with graduated sanctions. The framework treats secrecy and openness as endpoints of a spectrum, with controlled review (via a federated colloquium of qualified secure-review hosts) covering claims whose artefacts cannot be released publicly, and right-scaling claims whose artefacts cannot be reviewed even confidentially. The standard the community applies to its most consequential claims should be at least as high as the standard it applies to its least.

Related papers

• Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering [78.51705689800838]
  We propose an assurance-revision loop that uses deployment evidence to revise the requirements, model, or evaluator.<n>We then cast assured agentic development as a resource-allocation problem over human judgment, agent capability, and compute.
  arXiv  Detail & Related papers  (2026-09-10T17:58:48Z)
• AI Transparency: Governance Compliance or Stakeholder Requirements? [1.886618389105112]
  This paper presents an empirical analysis of 92 publicly available AI transparency statements published by Australian Government agencies.<n>Using a structured rubric derived from the mandated criteria, we evaluate how both the mandate and published statements are calibrated to each stakeholder class.
  arXiv  Detail & Related papers  (2026-06-16T03:13:34Z)
• What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents [2.8340200313563773]
  Benchmarks for autonomous agents measure whether agents complete tasks, yet this framing is systematically blind to whether an agent should have proceeded at all.<n>Agents trained under human-feedback objectives develop a structural tendency to proceed even when they lack the inputs, evidence, or authorization to act safely.<n>We introduce a three-gap taxonomy of abstention-warranted scenarios, covering specification gaps where required information is absent, verification gaps where world state cannot be confirmed, and authority gaps where explicit authorization has not been given.
  arXiv  Detail & Related papers  (2026-06-01T23:52:56Z)
• Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands [4.125187280299246]
  We argue that behavioural assurance, even when carefully designed, is being asked to carry safety claims it cannot verify.<n>We formalize this structural mismatch as the audit gap, the divergence between required and achievable verification access, and introduce the concept of fragile assurance.<n>We propose a technical pivot: bounding the weight of behavioral evidence in legal text and extending voluntary pre-deployment access with mechanistic-evidence classes.
  arXiv  Detail & Related papers  (2026-05-14T17:54:28Z)
• RISED: A Pre-Deployment Safety Evaluation Framework for Clinical AI Decision-Support Systems [0.0]
  We propose a five-dimension pre-deployment evaluation covering Reliability, Inclusivity, Sensitivity, Equity, and Deployability.<n> RISED is released as an open-source Python package that supplies the quantitative verdicts existing clinical AI reporting standards require.
  arXiv  Detail & Related papers  (2026-05-13T02:17:13Z)
• Bayesian Modeling for Uncertainty Management in Financial Risk Forecasting and Compliance [0.0]
  We develop an integrated approach that consistently enhances the handling of risk in market volatility forecasting, fraud detection, and compliance monitoring.<n>We evaluate the performance of one-day-ahead 95% Value-at-Risk (VaR) forecasts on daily S&P 500 returns, with a training period from 2000 to 2019 and an out-of-sample test period spanning 2020 to 2024.<n>Our proposed discount-factor DLM model produces a slightly liberal VaR estimate, with evidence of clustered violations.
  arXiv  Detail & Related papers  (2025-12-06T23:00:19Z)
• Prudential Reliability of Large Language Models in Reinsurance: Governance, Assurance, and Capital Efficiency [0.0]
  This paper develops a prudential framework for assessing the reliability of large language models (LLMs) in reinsurance.<n>A five-pillar architecture--governance, data lineage, assurance, resilience, and regulatory alignment--translates supervisory expectations from Solvency II, SR 11-7.<n>The framework is implemented through the Reinsurance AI Reliability and Assurance Benchmark (RAIRAB)
  arXiv  Detail & Related papers  (2025-11-11T10:33:54Z)
• Judging by the Rules: Compliance-Aligned Framework for Modern Slavery Statement Monitoring [24.13989765643719]
  Modern slavery affects millions of people worldwide, and regulatory frameworks such as Modern Slavery Acts now require companies to publish detailed disclosures.<n>These statements are often vague and inconsistent, making manual review time-consuming and difficult to scale.<n>We propose a novel framework that harnesses AI for rule-level compliance verification while preserving expert oversight.
  arXiv  Detail & Related papers  (2025-11-11T03:41:44Z)
• Zero-shot reasoning for simulating scholarly peer-review [0.0]
  We investigate a deterministic simulation framework that provides the first stable, evidence-based standard for evaluating AI-generated peer review reports.<n>First, the system is able to simulate calibrated editorial judgment, with 'Revise' decisions consistently forming the majority outcome.<n>Second, it maintains unwavering procedural integrity, enforcing a stable 29% evidence-anchoring compliance rate.
  arXiv  Detail & Related papers  (2025-10-02T13:59:14Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention [65.47632669243657]
  A dishonest institution can exploit mechanisms to discriminate or unjustly deny services under the guise of uncertainty.<n>We demonstrate the practicality of this threat by introducing an uncertainty-inducing attack called Mirage.<n>We propose Confidential Guardian, a framework that analyzes calibration metrics on a reference dataset to detect artificially suppressed confidence.
  arXiv  Detail & Related papers  (2025-05-29T19:47:50Z)
• SConU: Selective Conformal Uncertainty in Large Language Models [59.25881667640868]
  We propose a novel approach termed Selective Conformal Uncertainty (SConU)<n>We develop two conformal p-values that are instrumental in determining whether a given sample deviates from the uncertainty distribution of the calibration set at a specific manageable risk level.<n>Our approach not only facilitates rigorous management of miscoverage rates across both single-domain and interdisciplinary contexts, but also enhances the efficiency of predictions.
  arXiv  Detail & Related papers  (2025-04-19T03:01:45Z)
• AI Companies Should Report Pre- and Post-Mitigation Safety Evaluations [5.984437476321095]
  frontier AI companies should report both pre- and post-mitigation safety evaluations.<n> evaluating models at both stages provides policymakers with essential evidence to regulate deployment, access, and safety standards.
  arXiv  Detail & Related papers  (2025-03-17T17:56:43Z)
• Deliberative Alignment: Reasoning Enables Safer Language Models [64.60765108418062]
  We introduce Deliberative Alignment, a new paradigm that teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering.<n>We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers.
  arXiv  Detail & Related papers  (2024-12-20T21:00:11Z)
• From Mean to Extreme: Formal Differential Privacy Bounds on the Success of Real-World Data Reconstruction Attacks [54.25638567385662]
  Differential Privacy in machine learning is often interpreted as guarantees against membership inference.<n> translating DP budgets into quantitative protection against the more damaging threat of data reconstruction remains a challenging open problem.<n>This paper bridges the critical gap by deriving the first formal privacy bounds tailored to the mechanics of demonstrated "from-scratch" attacks.
  arXiv  Detail & Related papers  (2024-02-20T09:52:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.