Optimal Guarantees for Auditing Rényi Differentially Private Machine Learning

• URL: http://arxiv.org/abs/2605.21938v1
• Date: Thu, 21 May 2026 03:18:09 GMT
• Title: Optimal Guarantees for Auditing Rényi Differentially Private Machine Learning
• Abstract summary: We study black-box auditing for machine learning algorithms that claim R 'enyi differential privacy (RDP) guarantees.<n>We introduce an auditing framework, based on hypothesis testing, that directly estimates Rényi divergence between neighboring executions using the Donsker-Varadhan (DV) variational estimator.<n>Our analysis yields explicit and non-asymptotic confidence intervals for auditing via class-restricted DV estimators, separating statistical estimation error from algorithmic privacy leakage.
• Score: 15.701798841393668
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study black-box auditing for machine learning algorithms that claim R \ 'enyi differential privacy (RDP) guarantees. We introduce an auditing framework, based on hypothesis testing, that directly estimates Rényi divergence between neighboring executions using the Donsker-Varadhan (DV) variational estimator. Our analysis yields explicit and non-asymptotic confidence intervals for RDP auditing via class-restricted DV estimators, separating statistical estimation error from algorithmic privacy leakage. We prove matching minimax lower bounds showing that, up to logarithmic factors, our sample-complexity guarantees are information-theoretically optimal, thereby establishing the first optimal guarantees for auditing RDP via DV estimators. Empirically, we instantiate our framework for auditing DP-SGD in a fully black-box setting. Across MNIST and CIFAR-10, and over a wide range of privacy regimes, our auditors produce a strong overall improvement on empirical RDP lower bounds compared to prior state-of-the-art black-box methods especially at small and moderate Rényi orders where accurate auditing is most challenging.

Related papers

• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• Sequential Auditing for f-Differential Privacy [5.7992233755396505]
  We present new auditors to assess Differential Privacy (DP) of an algorithm based on output samples.<n>We shift the focus to the highly expressive privacy concept of $f$-DP, in which the entire privacy behavior is captured by a single tradeoff curve.
  arXiv  Detail & Related papers  (2026-02-06T09:22:24Z)
• KV-Auditor: Auditing Local Differential Privacy for Correlated Key-Value Estimation [3.1960143210470973]
  We propose KV-Auditor, a framework for auditing LDP-based key-value estimation mechanisms.<n>We classify state-of-the-art LDP key-value mechanisms into interactive and non-interactive types.<n>For interactive mechanisms, we design a segmentation strategy to capture incremental privacy leakage across iterations.
  arXiv  Detail & Related papers  (2025-08-15T14:17:24Z)
• UniAud: A Unified Auditing Framework for High Auditing Power and Utility with One Training Run [9.400936999321415]
  We propose a unified framework, UniAud, for data-independent auditing.<n>We then extend this framework as UniAud++ for data-dependent auditing.<n>We show that our framework matches the state-of-the-art auditing results of O(T) auditing with thousands of runs.
  arXiv  Detail & Related papers  (2025-07-06T16:35:48Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Enhancing One-run Privacy Auditing with Quantile Regression-Based Membership Inference [22.843200081364873]
  Differential privacy (DP) auditing aims to provide empirical lower bounds on the privacy guarantees of DP mechanisms like DP-SGD.<n>Recent work introduces one-run auditing approaches that effectively audit DP-SGD in white-box settings while still being computationally efficient.<n>In this work, we study how incorporating approaches for stronger membership inference attacks (MIA) can improve one-run auditing in the black-box setting.
  arXiv  Detail & Related papers  (2025-06-18T11:03:39Z)
• Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge [90.8674158031845]
  We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses.<n>This process effectively guides LLM-as-a-Judge to provide a more detailed chain-of-thought (CoT) judgment.<n>Our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling.
  arXiv  Detail & Related papers  (2025-02-18T03:31:06Z)
• Auditing Differential Privacy Guarantees Using Density Estimation [3.830092569453011]
  We present a novel method for accurately auditing the differential privacy guarantees of DP mechanisms. In particular, our solution is applicable to auditing DP guarantees of machine learning (ML) models.
  arXiv  Detail & Related papers  (2024-06-07T10:52:15Z)
• A Randomized Approach for Tight Privacy Accounting [63.67296945525791]
  We propose a new differential privacy paradigm called estimate-verify-release (EVR) EVR paradigm first estimates the privacy parameter of a mechanism, then verifies whether it meets this guarantee, and finally releases the query output. Our empirical evaluation shows the newly proposed EVR paradigm improves the utility-privacy tradeoff for privacy-preserving machine learning.
  arXiv  Detail & Related papers  (2023-04-17T00:38:01Z)
• Tight Auditing of Differentially Private Machine Learning [77.38590306275877]
  For private machine learning, existing auditing mechanisms are tight. They only give tight estimates under implausible worst-case assumptions. We design an improved auditing scheme that yields tight privacy estimates for natural (not adversarially crafted) datasets.
  arXiv  Detail & Related papers  (2023-02-15T21:40:33Z)
• Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations [50.37808220291108]
  This paper addresses learning safe output feedback control laws from partial observations of expert demonstrations. We first propose robust output control barrier functions (ROCBFs) as a means to guarantee safety. We then formulate an optimization problem to learn ROCBFs from expert demonstrations that exhibit safe system behavior.
  arXiv  Detail & Related papers  (2021-11-18T23:21:00Z)
• Privacy Preserving Recalibration under Domain Shift [119.21243107946555]
  We introduce a framework that abstracts out the properties of recalibration problems under differential privacy constraints. We also design a novel recalibration algorithm, accuracy temperature scaling, that outperforms prior work on private datasets.
  arXiv  Detail & Related papers  (2020-08-21T18:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.