Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems

• URL: http://arxiv.org/abs/2503.01470v1
• Date: Mon, 03 Mar 2025 12:24:59 GMT
• Title: Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems
• Authors: Ben Bucknall, Robert F. Trager, Michael A. Osborne,
• Abstract summary: External evaluation of AI systems is increasingly recognised as a crucial approach for understanding their potential risks.<n>Facilitating external evaluation in practice faces significant challenges in balancing evaluators' need for system access with AI developers' privacy and security concerns.
• Score: 17.53028680356076
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The external evaluation of AI systems is increasingly recognised as a crucial approach for understanding their potential risks. However, facilitating external evaluation in practice faces significant challenges in balancing evaluators' need for system access with AI developers' privacy and security concerns. Additionally, evaluators have reason to protect their own privacy - for example, in order to maintain the integrity of held-out test sets. We refer to the challenge of ensuring both developers' and evaluators' privacy as one of providing mutual privacy. In this position paper, we argue that (i) addressing this mutual privacy challenge is essential for effective external evaluation of AI systems, and (ii) current methods for facilitating external evaluation inadequately address this challenge, particularly when it comes to preserving evaluators' privacy. In making these arguments, we formalise the mutual privacy problem; examine the privacy and access requirements of both model owners and evaluators; and explore potential solutions to this challenge, including through the application of cryptographic and hardware-based approaches.

Related papers

• Securing External Deeper-than-black-box GPAI Evaluations [49.1574468325115]
  This paper examines the critical challenges and potential solutions for conducting secure and effective external evaluations of general-purpose AI (GPAI) models. With the exponential growth in size, capability, reach and accompanying risk, ensuring accountability, safety, and public trust requires frameworks that go beyond traditional black-box methods.
  arXiv  Detail & Related papers  (2025-03-10T16:13:45Z)
• Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curators [11.819135112719623]
  We argue that despite potential advantages in addressing contamination issues, private evaluations introduce inadvertent financial and evaluation risks. Key concerns include the potential conflict of interest arising from private data curators' business relationships with their clients. We highlight that the subjective preferences of private expert annotators will lead to inherent evaluation bias towards the models trained with the private curators' data.
  arXiv  Detail & Related papers  (2025-02-09T23:57:33Z)
• Toward Ethical AI: A Qualitative Analysis of Stakeholder Perspectives [0.0]
  This study explores stakeholder perspectives on privacy in AI systems, focusing on educators, parents, and AI professionals.<n>Using qualitative analysis of survey responses from 227 participants, the research identifies key privacy risks, including data breaches, ethical misuse, and excessive data collection.<n>The findings provide actionable insights into balancing the benefits of AI with robust privacy protections.
  arXiv  Detail & Related papers  (2025-01-23T02:06:25Z)
• Privacy Risks of General-Purpose AI Systems: A Foundation for Investigating Practitioner Perspectives [47.17703009473386]
  Powerful AI models have led to impressive leaps in performance across a wide range of tasks. Privacy concerns have led to a wealth of literature covering various privacy risks and vulnerabilities of AI models. We conduct a systematic review of these survey papers to provide a concise and usable overview of privacy risks in GPAIS.
  arXiv  Detail & Related papers  (2024-07-02T07:49:48Z)
• Centering Policy and Practice: Research Gaps around Usable Differential Privacy [12.340264479496375]
  We argue that while differential privacy is a clean formulation in theory, it poses significant challenges in practice. To bridge the gaps between differential privacy's promises and its real-world usability, researchers and practitioners must work together.
  arXiv  Detail & Related papers  (2024-06-17T21:32:30Z)
• Experts-in-the-Loop: Establishing an Effective Workflow in Crafting Privacy Q&A [0.0]
  We propose a dynamic workflow for transforming privacy policies into privacy question-and-answer (Q&A) pairs. Thereby, we facilitate interdisciplinary collaboration among legal experts and conversation designers. Our proposed workflow underscores continuous improvement and monitoring throughout the construction of privacy Q&As.
  arXiv  Detail & Related papers  (2023-11-18T20:32:59Z)
• Technocracy, pseudoscience and performative compliance: the risks of privacy risk assessments. Lessons from NIST's Privacy Risk Assessment Methodology [0.0]
  Privacy risk assessments have been touted as an objective, principled way to encourage organizations to implement privacy-by-design. Existing guidelines and methods remain vague, and there is little empirical evidence on privacy harms. We highlight the limitations and pitfalls of what is essentially a utilitarian and technocratic approach.
  arXiv  Detail & Related papers  (2023-08-24T01:32:35Z)
• Auditing and Generating Synthetic Data with Controllable Trust Trade-offs [54.262044436203965]
  We introduce a holistic auditing framework that comprehensively evaluates synthetic datasets and AI models. It focuses on preventing bias and discrimination, ensures fidelity to the source data, assesses utility, robustness, and privacy preservation. We demonstrate the framework's effectiveness by auditing various generative models across diverse use cases.
  arXiv  Detail & Related papers  (2023-04-21T09:03:18Z)
• A Randomized Approach for Tight Privacy Accounting [63.67296945525791]
  We propose a new differential privacy paradigm called estimate-verify-release (EVR) EVR paradigm first estimates the privacy parameter of a mechanism, then verifies whether it meets this guarantee, and finally releases the query output. Our empirical evaluation shows the newly proposed EVR paradigm improves the utility-privacy tradeoff for privacy-preserving machine learning.
  arXiv  Detail & Related papers  (2023-04-17T00:38:01Z)
• Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deployment [100.1798289103163]
  We present a detailed review of current practices and state-of-the-art methodologies in the field of differential privacy (DP) Key points and high-level contents of the article were originated from the discussions from "Differential Privacy (DP): Challenges Towards the Next Frontier" This article aims to provide a reference point for the algorithmic and design decisions within the realm of privacy, highlighting important challenges and potential research directions.
  arXiv  Detail & Related papers  (2023-04-14T05:29:18Z)
• Differential Privacy and Fairness in Decisions and Learning Tasks: A Survey [50.90773979394264]
  It reviews the conditions under which privacy and fairness may have aligned or contrasting goals. It analyzes how and why DP may exacerbate bias and unfairness in decision problems and learning tasks.
  arXiv  Detail & Related papers  (2022-02-16T16:50:23Z)
• Trustworthy AI [75.99046162669997]
  Brittleness to minor adversarial changes in the input data, ability to explain the decisions, address the bias in their training data, are some of the most prominent limitations. We propose the tutorial on Trustworthy AI to address six critical issues in enhancing user and public trust in AI systems.
  arXiv  Detail & Related papers  (2020-11-02T20:04:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.