Algorithmic audits of algorithms, and the law

• URL: http://arxiv.org/abs/2203.03711v1
• Date: Tue, 15 Feb 2022 14:20:53 GMT
• Title: Algorithmic audits of algorithms, and the law
• Authors: Erwan Le Merrer and Ronan Pons and Gilles Tr\'edan
• Abstract summary: We focus on external audits that are conducted by interacting with the user side of the target algorithm. The legal framework in which these audits take place is mostly ambiguous to researchers developing them. This article highlights the relation of current audits with law, in order to structure the growing field of algorithm auditing.
• Score: 3.9103337761169943
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Algorithmic decision making is now widespread, ranging from health care allocation to more common actions such as recommendation or information ranking. The aim to audit these algorithms has grown alongside. In this paper, we focus on external audits that are conducted by interacting with the user side of the target algorithm, hence considered as a black box. Yet, the legal framework in which these audits take place is mostly ambiguous to researchers developing them: on the one hand, the legal value of the audit outcome is uncertain; on the other hand the auditors' rights and obligations are unclear. The contribution of this paper is to articulate two canonical audit forms to law, to shed light on these aspects: 1) the first audit form (we coin the Bobby audit form) checks a predicate against the algorithm, while the second (Sherlock) is more loose and opens up to multiple investigations. We find that: Bobby audits are more amenable to prosecution, yet are delicate as operating on real user data. This can lead to reject by a court (notion of admissibility). Sherlock audits craft data for their operation, most notably to build surrogates of the audited algorithm. It is mostly used for acts for whistleblowing, as even if accepted as a proof, the evidential value will be low in practice. 2) these two forms require the prior respect of a proper right to audit, granted by law or by the platform being audited; otherwise the auditor will be also prone to prosecutions regardless of the audit outcome. This article thus highlights the relation of current audits with law, in order to structure the growing field of algorithm auditing.

Related papers

• Auditing for Bias in Ad Delivery Using Inferred Demographic Attributes [50.37313459134418]
  We study the effects of inference error on auditing for bias in one prominent application: black-box audit of ad delivery using paired ads. We propose a way to mitigate the inference error when evaluating skew in ad delivery algorithms.
  arXiv  Detail & Related papers  (2024-10-30T18:57:03Z)
• From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing [1.196505602609637]
  Audits can take many forms, including pre-deployment risk assessments, ongoing monitoring, and compliance testing. There are many operational challenges to AI auditing that complicate its implementation. We argue that auditing can be cast as a natural hypothesis test, draw parallels hypothesis testing and legal procedure, and argue that this framing provides clear and interpretable guidance on audit implementation.
  arXiv  Detail & Related papers  (2024-10-07T06:15:46Z)
• A Game-Theoretic Analysis of Auditing Differentially Private Algorithms with Epistemically Disparate Herd [16.10098472773814]
  This study examines the impact of herd audits on algorithm developers using the Stackelberg game approach. By enhancing transparency and accountability, herd audit contributes to the responsible development of privacy-preserving algorithms.
  arXiv  Detail & Related papers  (2024-04-24T20:34:27Z)
• The Decisive Power of Indecision: Low-Variance Risk-Limiting Audits and Election Contestation via Marginal Mark Recording [51.82772358241505]
  Risk-limiting audits (RLAs) are techniques for verifying the outcomes of large elections. We define new families of audits that improve efficiency and offer advances in statistical power. New audits are enabled by revisiting the standard notion of a cast-vote record so that it can declare multiple possible mark interpretations.
  arXiv  Detail & Related papers  (2024-02-09T16:23:54Z)
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem [0.971392598996499]
  We provide the first comprehensive field scan of the AI audit ecosystem. We identify emerging best practices as well as methods and tools that are becoming commonplace. We outline policy recommendations to improve the quality and impact of these audits.
  arXiv  Detail & Related papers  (2023-10-04T01:40:03Z)
• On the relevance of APIs facing fairwashed audits [3.479254848034425]
  Recent legislation required AI platforms to provide APIs for regulators to assess their compliance with the law. Research has shown that platforms can manipulate their API answers through fairwashing. This paper studies the benefits of the joint use of platform scraping and of APIs.
  arXiv  Detail & Related papers  (2023-05-23T10:06:22Z)
• The right to audit and power asymmetries in algorithm auditing [68.8204255655161]
  We elaborate on the challenges and asymmetries mentioned by Sandvig at the IC2S2 2021. We also contribute a discussion of the asymmetries that were not covered by Sandvig. We discuss the implications these asymmetries have for algorithm auditing research.
  arXiv  Detail & Related papers  (2023-02-16T13:57:41Z)
• Tight Auditing of Differentially Private Machine Learning [77.38590306275877]
  For private machine learning, existing auditing mechanisms are tight. They only give tight estimates under implausible worst-case assumptions. We design an improved auditing scheme that yields tight privacy estimates for natural (not adversarially crafted) datasets.
  arXiv  Detail & Related papers  (2023-02-15T21:40:33Z)
• Having your Privacy Cake and Eating it Too: Platform-supported Auditing of Social Media Algorithms for Public Interest [70.02478301291264]
  Social media platforms curate access to information and opportunities, and so play a critical role in shaping public discourse. Prior studies have used black-box methods to show that these algorithms can lead to biased or discriminatory outcomes. We propose a new method for platform-supported auditing that can meet the goals of the proposed legislation.
  arXiv  Detail & Related papers  (2022-07-18T17:32:35Z)
• Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models [73.24381010980606]
  This study examines issues of algorithmic fairness in the context of systems that inform tax audit selection by the IRS. We show how the use of more flexible machine learning methods for selecting audits may affect vertical equity. Our results have implications for the design of algorithmic tools across the public sector.
  arXiv  Detail & Related papers  (2022-06-20T16:27:06Z)
• Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits [0.0]
  This review follows PRISMA guidelines in a review of over 500 English articles that yielded 62 algorithm audit studies. The studies are synthesized and organized primarily by behavior (discrimination, distortion, exploitation, and misjudgement) The paper concludes by offering the common ingredients of successful audits, and discussing algorithm auditing in the context of broader research.
  arXiv  Detail & Related papers  (2021-02-03T19:21:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.