SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue

• URL: http://arxiv.org/abs/2609.01548v1
• Date: Tue, 01 Sep 2026 17:15:22 GMT
• Title: SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue
• Abstract summary: We introduce SDARE-Bench, the first scenario-based benchmark evaluating stigma detection and open-ended response generation in large language models.<n> Empirical results show poor identification of stigma components, especially in group dialogues.<n>Our findings identify stigma response as a recurring LLM safety vulnerability, especially in socially complex conversational contexts.
• Score: 23.532923483545076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) are increasingly used in advice seeking and decision making that may affect social judgements. Despite stigma's profound effects on people and communities, benchmarks remain scarce. Existing general-domain evaluations typically rely on static prompts and fixed-format tasks, overlooking conversational contexts and audience effects in everyday communication. To address these gaps, we introduce SDARE-Bench, the first scenario-based benchmark evaluating both stigma detection and open-ended response generation in LLMs, comprising 1,138 dyadic queries and 1,388 group dialogue. Empirical results across 8 LLMs consistently demonstrate poor identification of stigma components, especially in group dialogues. In open-ended response generation, stigma expression was substantially higher in group settings than in dyadic, with weaker resistance to stigma and more unrealistic advice. Responses were evaluated using a classifier trained on 1,392 human annotated responses. In constructed group pressure settings, stigma expression rates further increased to a striking average of 97.5%. Our findings identify stigma response as a recurring LLM safety vulnerability, especially in socially complex conversational contexts.

Related papers

• Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts [4.801119915366712]
  Large language models (LLMs) used for distress detection are typically aligned to a single, undifferentiated standard.<n>How well do these models capture the perspectives of the communities whose language they assess?<n>We evaluate nine open-weight LLM configurations and four frontier configurations against these labels.
  arXiv  Detail & Related papers  (2026-08-29T21:23:57Z)
• Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthromorphism, and Maxims [54.31707479192195]
  Large language models (LLMs) are being increasingly used to answer subjective, information-seeking questions.<n>Existing evaluations for subjective cultural queries largely focus on factual correctness, ignoring how the response is framed.<n>We introduce FRANZ, an automated FRAmework for respoNse characteriZation to conduct communicative audit of LLM responses along four dimensions.
  arXiv  Detail & Related papers  (2026-06-01T17:01:31Z)
• Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL [64.3268313484078]
  Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare.<n>Their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns.<n>We investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception.
  arXiv  Detail & Related papers  (2025-10-16T05:29:36Z)
• Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses [2.3112192919085826]
  Large Language Models (LLMs) are increasingly used as proxies for human subjects in social science surveys.<n>Their reliability and susceptibility to known human-like response biases are poorly understood.<n>This work investigates the response robustness of LLMs in normative survey contexts.
  arXiv  Detail & Related papers  (2025-07-09T18:01:50Z)
• Information Anxiety in Large Language Models [21.574677910096735]
  Large Language Models (LLMs) have demonstrated strong performance as knowledge repositories. We take the investigation further by conducting a comprehensive analysis of the internal reasoning and retrieval mechanisms of LLMs. Our work focuses on three critical dimensions - the impact of entity popularity, the models' sensitivity to lexical variations in query formulation, and the progression of hidden state representations.
  arXiv  Detail & Related papers  (2024-11-16T14:28:33Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• OR-Bench: An Over-Refusal Benchmark for Large Language Models [65.34666117785179]
  Large Language Models (LLMs) require careful safety alignment to prevent malicious outputs.<n>This study proposes a novel method for automatically generating large-scale over-refusal datasets.<n>We introduce OR-Bench, the first large-scale over-refusal benchmark.
  arXiv  Detail & Related papers  (2024-05-31T15:44:33Z)
• Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models [51.75805497456226]
  This work focuses on the factual consistency issue with the help of the dialogue summarization task. Our evaluation shows that, on average, 26.8% of the summaries generated by LLMs contain factual inconsistency. To stimulate and enhance the dialogue comprehension ability of LLMs, we propose a fine-tuning paradigm with auto-constructed multi-task data.
  arXiv  Detail & Related papers  (2023-11-13T09:32:12Z)
• Evaluate What You Can't Evaluate: Unassessable Quality for Generated Response [56.25966921370483]
  There are challenges in using reference-free evaluators based on large language models. Reference-free evaluators are more suitable for open-ended examples with different semantics responses. There are risks in using eference-free evaluators based on LLMs to evaluate the quality of dialogue responses.
  arXiv  Detail & Related papers  (2023-05-24T02:52:48Z)
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms [41.82685006832153]
  We propose a new challenge, RICA: Robust Inference capability based on Commonsense Axioms. We generate data for this challenge using commonsense knowledge bases and probe PTLMs across two different evaluation settings. Experiments show that PTLMs perform no better than random guessing on the zero-shot setting, are heavily impacted by statistical biases, and are not robust to perturbation attacks.
  arXiv  Detail & Related papers  (2020-05-02T10:36:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.