Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers

• URL: http://arxiv.org/abs/2501.13302v1
• Date: Thu, 23 Jan 2025 01:04:00 GMT
• Title: Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers
• Authors: Akshit Achara, Anshuman Chhabra,
• Abstract summary: Safety Moderation (ASM) classifiers are designed to moderate content on social media platforms. It is crucial to ensure that these classifiers do not unfairly classify content belonging to users from minority groups. We thus examine the fairness and robustness of four widely-used, closed-source ASM classifiers.
• Score: 5.35599092568615
• License:
• Abstract: AI Safety Moderation (ASM) classifiers are designed to moderate content on social media platforms and to serve as guardrails that prevent Large Language Models (LLMs) from being fine-tuned on unsafe inputs. Owing to their potential for disparate impact, it is crucial to ensure that these classifiers: (1) do not unfairly classify content belonging to users from minority groups as unsafe compared to those from majority groups and (2) that their behavior remains robust and consistent across similar inputs. In this work, we thus examine the fairness and robustness of four widely-used, closed-source ASM classifiers: OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL) API, and Clarifai API. We assess fairness using metrics such as demographic parity and conditional statistical parity, comparing their performance against ASM models and a fair-only baseline. Additionally, we analyze robustness by testing the classifiers' sensitivity to small and natural input perturbations. Our findings reveal potential fairness and robustness gaps, highlighting the need to mitigate these issues in future versions of these models.

Related papers

• Smoothed Embeddings for Robust Language Models [11.97873981355746]
  Large language models (LLMs) are vulnerable to jailbreaking attacks that subvert alignment and induce harmful outputs. We propose the Randomized Embedding Smoothing and Token Aggregation (RESTA) defense, which adds random noise to the embedding vectors and performs aggregation during the generation of each output token. Our experiments demonstrate that our approach achieves superior robustness versus utility tradeoffs compared to the baseline defenses.
  arXiv  Detail & Related papers  (2025-01-27T20:57:26Z)
• Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble [2.1450827490014865]
  We present a light-weight, post-processing method for mitigating counterfactual fairness in closed-source text safety classifiers. We introduce two threshold-agnostic metrics to assess the counterfactual fairness of a model, and demonstrate how combining these metrics with Fair Data Reweighting (FDW) helps mitigate biases. Our results show that our approach improves counterfactual fairness with minimal impact on model performance.
  arXiv  Detail & Related papers  (2024-09-05T14:35:35Z)
• Representation Magnitude has a Liability to Privacy Vulnerability [3.301728339780329]
  We propose a plug-in model-level solution to mitigate membership privacy leakage. Our approach ameliorates models' privacy vulnerability while maintaining generalizability.
  arXiv  Detail & Related papers  (2024-07-23T04:13:52Z)
• Advancing the Robustness of Large Language Models through Self-Denoised Smoothing [50.54276872204319]
  Large language models (LLMs) have achieved significant success, but their vulnerability to adversarial perturbations has raised considerable concerns. We propose to leverage the multitasking nature of LLMs to first denoise the noisy inputs and then to make predictions based on these denoised versions. Unlike previous denoised smoothing techniques in computer vision, which require training a separate model to enhance the robustness of LLMs, our method offers significantly better efficiency and flexibility.
  arXiv  Detail & Related papers  (2024-04-18T15:47:00Z)
• Characterizing Data Point Vulnerability via Average-Case Robustness [29.881355412540557]
  adversarial robustness is a standard framework, which views robustness of predictions through a binary lens. We consider a complementary framework for robustness, called average-case robustness, which measures the fraction of points in a local region. We show empirically that our estimators are accurate and efficient for standard deep learning models.
  arXiv  Detail & Related papers  (2023-07-26T01:10:29Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• User-Centered Security in Natural Language Processing [0.7106986689736825]
  dissertation proposes a framework of user-centered security in Natural Language Processing (NLP) It focuses on two security domains within NLP with great public interest.
  arXiv  Detail & Related papers  (2023-01-10T22:34:19Z)
• Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach [131.20444904674494]
  We tackle the problem of measuring group fairness under unawareness of sensitive attributes. We show that quantification approaches are particularly suited to tackle the fairness-under-unawareness problem.
  arXiv  Detail & Related papers  (2021-09-17T13:45:46Z)
• Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation [74.05906222376608]
  We propose adversarial self-supervision UDA (or ASSUDA) that maximizes the agreement between clean images and their adversarial examples by a contrastive loss in the output space. This paper is rooted in two observations: (i) the robustness of UDA methods in semantic segmentation remains unexplored, which pose a security concern in this field; and (ii) although commonly used self-supervision (e.g., rotation and jigsaw) benefits image tasks such as classification and recognition, they fail to provide the critical supervision signals that could learn discriminative representation for segmentation tasks.
  arXiv  Detail & Related papers  (2021-05-23T01:50:44Z)
• RobustBench: a standardized adversarial robustness benchmark [84.50044645539305]
  Key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks. We analyze the impact of robustness on the performance on distribution shifts, calibration, out-of-distribution detection, fairness, privacy leakage, smoothness, and transferability.
  arXiv  Detail & Related papers  (2020-10-19T17:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.