Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback

• URL: http://arxiv.org/abs/2609.13714v1
• Date: Sat, 12 Sep 2026 05:00:50 GMT
• Title: Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback
• Abstract summary: We study checkpoint selection subject to non-regression tolerances relative to a retained incumbent.<n>We give a reproducible release procedure that separates candidate search from independent, paired evaluation.<n>A joint release decision does not require a slice-count Bonferroni penalty, although certification power can still decrease with the number of slices.
• Score: 2.8716825293247665
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An updated model can improve an aggregate metric while degrading a slice that matters to a downstream user. We study checkpoint selection subject to non-regression tolerances relative to a retained incumbent. The central distinction is between failing to detect harm and certifying non-inferiority: the former can release harmful updates with high probability when evaluation is noisy. We give a reproducible release procedure that separates candidate search from independent, paired evaluation and returns the exact incumbent when certification fails. Applying established intersection-union and Learn-then-Test principles, we state finite-sample guarantees for one frozen candidate, a finite candidate library, and a prespecified testing order. A joint release decision does not require a slice-count Bonferroni penalty, although certification power can still decrease with the number of slices. In bounded-score simulations, a no-detected-harm gate releases a harmful candidate in 99.7% of trials in one 32-slice setting, compared with 2.6% for an exact non-inferiority gate at a 5% target. A constructed two-block family yields larger certified utility than a scalar path under matched candidate counts. Public digits experiments, including a subsequent continuation that improves average aggregate accuracy, return the incumbent in every run because certification is underpowered. These results establish an auditable protocol and its limitations; they do not establish benefits on foundation-model or multilingual translation upgrades.

Related papers

• The Order Is the Guarantee: Verifier-Budgeted Code Deletion with Static-First Learned Proposals [12.095189927690988]
  We study the inverse problem-how an Al system should remove code when execution-verification capacity is finite.<n>We formulate redundant-code reduction as proposal scheduling.<n>We show that scheduling governs search while the test suite alone governs what "preserving behavior" means.
  arXiv  Detail & Related papers  (2026-08-05T09:16:29Z)
• Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents [1.4915002678801434]
  We show that Bonferroni allocation is distribution-free but conservative under correlated errors.<n> coarse-to-fine label selection can create near-perfect measured correlation without learned dependence.<n>A modular certificate using per-stage certificates and a pairwise overlap bound yields a positive average gain of 0.6%.
  arXiv  Detail & Related papers  (2026-08-04T23:48:36Z)
• What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents [0.0]
  guardrails act before irreversible tool calls, but their guarantees depend on what policy state is representable.<n>Pearson representation attacks add a margin, so benign calibration alone does not transfer.<n>Experiments target these distinctions through static diagnostics, controlled-model enumeration, representation rewrites, and paired closed-loop reruns.
  arXiv  Detail & Related papers  (2026-07-24T19:14:26Z)
• MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning [19.054149750597933]
  MIST-RL (Mutation-based Incremental Suite Testing via Reinforcement Learning) is a framework that shifts the focus to "scaling-by-utility"<n>We introduce a novel incremental mutation reward combined with dynamic penalties, which incentivizes the model to discover new faults while it suppresses functionally equivalent assertions.<n>Experiments on HumanEval+ and MBPP+ demonstrate that MIST-RL outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T03:22:44Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)
• (Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy [8.010528849585937]
  We derive an (almost) guaranteed upper bound on the error of deep neural networks under distribution shift using unlabeled test data. In particular, our bound requires a simple, intuitive condition which is well justified by prior empirical works. We expect this loss can serve as a drop-in replacement for future methods which require maximizing multiclass disagreement.
  arXiv  Detail & Related papers  (2023-06-01T03:22:15Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.