Discriminatory Compliance: How LLMs Answer Queries from Protected Groups

• URL: http://arxiv.org/abs/2606.21296v1
• Date: Fri, 19 Jun 2026 10:19:54 GMT
• Title: Discriminatory Compliance: How LLMs Answer Queries from Protected Groups
• Abstract summary: We show that state-of-the-art Large Language Models respond inconsistently to questions from protected identity groups.<n>We define discriminatory compliance as patterns in question answering that disproportionately disadvantage users from minority or protected backgrounds.
• Score: 0.9730476478298455
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Chatbots developed using Large Language Models (LLMs) implement various safeguards for sensitive questions and/or scenarios. These safeguards require making certain assumptions about the person asking the question. We define discriminatory compliance as patterns in question answering that disproportionately disadvantage users from minority or protected backgrounds, for instance by omitting information that would be valuable for them. In this paper, we show that state-of-the-art LLMs respond inconsistently to questions from personas from protected identity groups, and that some of these inconsistencies mean that key information that should be provided to minority or protected background personas is missing. We show that this behavior is, additionally, inconsistent across and within model providers as well as across background conditions and ways of phrasing those conditions.

Related papers

• You Shouldn't Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusals [6.358967433398892]
  We propose the first taxonomy of LLM refusals grounded in pragmatic theory.<n>Although models differ in how they refuse, their refusals are overall explicit and strongly morally evaluative.<n>We call for alignment evaluation that considers not only whether models refuse harmful requests, but also whether they refuse in ways that are contextually adaptive and socially accountable for the interactional consequences of saying no.
  arXiv  Detail & Related papers  (2026-08-31T14:19:16Z)
• Boundary-targeted Membership Inference Attacks on Safety Classifiers [70.20833439671131]
  Safety classifiers are trained on sensitive datasets including discussions of self-harm and mental health.<n>We introduce a new boundary-targeted selection strategy that identifies low confidence examples.<n>Experiments show that an adversary can recover 19% of the conversations a safety classifier flagged as indicating user distress, at a 5% false-positive rate.<n>This is $3.5$ times more than attacking using state-of-the-art MIA methods alone.
  arXiv  Detail & Related papers  (2026-05-21T12:05:22Z)
• Evaluating Language Model Reasoning about Confidential Information [95.64687778185703]
  We study whether language models exhibit contextual robustness, or the capability to adhere to context-dependent safety specifications.<n>We develop a benchmark (PasswordEval) that measures whether language models can correctly determine when a user request is authorized.<n>We find that current open- and closed-source models struggle with this seemingly simple task, and that, perhaps surprisingly, reasoning capabilities do not generally improve performance.
  arXiv  Detail & Related papers  (2025-08-27T15:39:46Z)
• Guarding Your Conversations: Privacy Gatekeepers for Secure Interactions with Cloud-Based AI Models [0.34998703934432673]
  We propose the concept of an "LLM gatekeeper", a lightweight, locally run model that filters out sensitive information from user queries before they are sent to the potentially untrustworthy, though highly capable, cloud-based LLM.<n>Through experiments with human subjects, we demonstrate that this dual-model approach introduces minimal overhead while significantly enhancing user privacy, without compromising the quality of LLM responses.
  arXiv  Detail & Related papers  (2025-08-22T19:49:03Z)
• SoK: Semantic Privacy in Large Language Models [24.99241770349404]
  This paper introduces a lifecycle-centric framework to analyze semantic privacy risks across input processing, pretraining, fine-tuning, and alignment stages of Large Language Models (LLMs)<n>We categorize key attack vectors and assess how current defenses, such as differential privacy, embedding encryption, edge computing, and unlearning, address these threats.<n>We conclude by outlining open challenges, including quantifying semantic leakage, protecting multimodal inputs, balancing de-identification with generation quality, and ensuring transparency in privacy enforcement.
  arXiv  Detail & Related papers  (2025-06-30T08:08:15Z)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? [83.53005932513155]
  Multi-modal large language models (MLLMs) have made significant progress, yet their safety alignment remains limited.<n>We propose finetuning MLLMs on a small set of benign instruct-following data with responses replaced by simple, clear rejection sentences.
  arXiv  Detail & Related papers  (2025-04-14T09:03:51Z)
• Do LLMs exhibit demographic parity in responses to queries about Human Rights? [4.186018120368565]
  Hedging and non-affirmation are behaviours that express ambiguity or a lack of clear endorsement on specific statements.<n>We design a novel prompt set on human rights in the context of different national or social identities.<n>We develop metrics to capture hedging and non-affirmation behaviours.<n>We find that all models exhibit some demographic disparities in how they attribute human rights between different identity groups.
  arXiv  Detail & Related papers  (2025-02-26T15:19:35Z)
• Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies [66.30619782227173]
  Large language models (LLMs) can produce erroneous responses that sound fluent and convincing.<n>We identify several features of LLM responses that shape users' reliance.<n>We find that explanations increase reliance on both correct and incorrect responses.<n>We observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies.
  arXiv  Detail & Related papers  (2025-02-12T16:35:41Z)
• Understanding the Relationship between Prompts and Response Uncertainty in Large Language Models [55.332004960574004]
  Large language models (LLMs) are widely used in decision-making, but their reliability, especially in critical tasks like healthcare, is not well-established.<n>This paper investigates how the uncertainty of responses generated by LLMs relates to the information provided in the input prompt.<n>We propose a prompt-response concept model that explains how LLMs generate responses and helps understand the relationship between prompts and response uncertainty.
  arXiv  Detail & Related papers  (2024-07-20T11:19:58Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• Can Querying for Bias Leak Protected Attributes? Achieving Privacy With Smooth Sensitivity [14.564919048514897]
  Existing regulations prohibit model developers from accessing protected attributes. We show that simply querying for fairness metrics can leak the protected attributes of individuals to the model developers. We propose Attribute-Conceal, a novel technique that achieves differential privacy by calibrating noise to the smooth sensitivity of our bias query.
  arXiv  Detail & Related papers  (2022-11-03T20:44:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.