Auditing for Bias in Ad Delivery Using Inferred Demographic Attributes

• URL: http://arxiv.org/abs/2410.23394v1
• Date: Wed, 30 Oct 2024 18:57:03 GMT
• Title: Auditing for Bias in Ad Delivery Using Inferred Demographic Attributes
• Authors: Basileal Imana, Aleksandra Korolova, John Heidemann,
• Abstract summary: We study the effects of inference error on auditing for bias in one prominent application: black-box audit of ad delivery using paired ads. We propose a way to mitigate the inference error when evaluating skew in ad delivery algorithms.
• Score: 50.37313459134418
• License:
• Abstract: Auditing social-media algorithms has become a focus of public-interest research and policymaking to ensure their fairness across demographic groups such as race, age, and gender in consequential domains such as the presentation of employment opportunities. However, such demographic attributes are often unavailable to auditors and platforms. When demographics data is unavailable, auditors commonly infer them from other available information. In this work, we study the effects of inference error on auditing for bias in one prominent application: black-box audit of ad delivery using paired ads. We show that inference error, if not accounted for, causes auditing to falsely miss skew that exists. We then propose a way to mitigate the inference error when evaluating skew in ad delivery algorithms. Our method works by adjusting for expected error due to demographic inference, and it makes skew detection more sensitive when attributes must be inferred. Because inference is increasingly used for auditing, our results provide an important addition to the auditing toolbox to promote correct audits of ad delivery algorithms for bias. While the impact of attribute inference on accuracy has been studied in other domains, our work is the first to consider it for black-box evaluation of ad delivery bias, when only aggregate data is available to the auditor.

Related papers

• Auditing for Racial Discrimination in the Delivery of Education Ads [50.37313459134418]
  We propose a new third-party auditing method that can evaluate racial bias in the delivery of ads for education opportunities. We find evidence of racial discrimination in Meta's algorithmic delivery of ads for education opportunities, posing legal and ethical concerns.
  arXiv  Detail & Related papers  (2024-06-02T02:00:55Z)
• The Impact of Differential Feature Under-reporting on Algorithmic Fairness [86.275300739926]
  We present an analytically tractable model of differential feature under-reporting. We then use to characterize the impact of this kind of data bias on algorithmic fairness. Our results show that, in real world data settings, under-reporting typically leads to increasing disparities.
  arXiv  Detail & Related papers  (2024-01-16T19:16:22Z)
• A Brief Tutorial on Sample Size Calculations for Fairness Audits [6.66743248310448]
  This tutorial provides guidance on how to determine the required subgroup sample sizes for a fairness audit. Our findings are applicable to audits of binary classification models and multiple fairness metrics derived as summaries of the confusion matrix.
  arXiv  Detail & Related papers  (2023-12-07T22:59:12Z)
• D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias [57.87117733071416]
  We propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases. A user can detect the presence of bias against a group by identifying unfair causal relationships in the causal network. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset.
  arXiv  Detail & Related papers  (2022-08-10T03:41:48Z)
• Understanding Unfairness in Fraud Detection through Model and Data Bias Interactions [4.159343412286401]
  We argue that algorithmic unfairness stems from interactions between models and biases in the data. We study a set of hypotheses regarding the fairness-accuracy trade-offs that fairness-blind ML algorithms exhibit under different data bias settings.
  arXiv  Detail & Related papers  (2022-07-13T15:18:30Z)
• Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models [73.24381010980606]
  This study examines issues of algorithmic fairness in the context of systems that inform tax audit selection by the IRS. We show how the use of more flexible machine learning methods for selecting audits may affect vertical equity. Our results have implications for the design of algorithmic tools across the public sector.
  arXiv  Detail & Related papers  (2022-06-20T16:27:06Z)
• Auditing for Discrimination in Algorithms Delivering Job Ads [70.02478301291264]
  We develop a new methodology for black-box auditing of algorithms for discrimination in the delivery of job advertisements. Our first contribution is to identify the distinction between skew in ad delivery due to protected categories such as gender or race. Second, we develop an auditing methodology that distinguishes between skew explainable by differences in qualifications from other factors. Third, we apply our proposed methodology to two prominent targeted advertising platforms for job ads: Facebook and LinkedIn.
  arXiv  Detail & Related papers  (2021-04-09T17:38:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.