Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed

• URL: http://arxiv.org/abs/2607.24898v1
• Date: Mon, 27 Jul 2026 16:42:36 GMT
• Title: Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed
• Abstract summary: State-of-the-art toxicity detectors for text-to-image generation adopt a one-size-fits-all approach.<n>Approximately 35% of generated images labeled safe are considered harmful by disability communities.<n>We argue for community-specific toxicity detection (CTD)
• Score: 14.067913369851894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art toxicity detectors for text-to-image generation adopt a one-size-fits-all approach: a single universal model applying fixed safety guidelines to all users. Our empirical evidence shows that these detectors fail to shield marginalized communities: approximately 35% of generated images labeled safe are considered harmful by disability communities. In this position paper, we argue for community-specific toxicity detection (CTD). To demonstrate its feasibility, we collaborate with disability experts to develop safety guidelines for two communities: dwarfism and blind/low vision. Using a dataset of 2,400 annotated T2I-generated images we demonstrate that both large vision-language models and existing general-purpose toxicity detectors catastrophically fail to recognize harmful content under these guidelines in zero-shot settings with F1 score lower than random guessing (F1 0.32 and 0.37). Promisingly, prompt-based adaptation methods (ICL, VQA) substantially improve harm detection performance (GPT-4o: F1 0.50 and 0.78), while parameter-efficient fine-tuning improves smaller models (0.5b-7b with best F1 0.48 and 0.59) with less than 100 demonstrations, but remains sensitive to evolving guidelines. Despite these gains, CTD performance remains far below F1 $\approx 0.9$ achieved for general-purpose toxicity detection, highlighting the challenge and the need for sustained research effort.

Related papers

• When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs [54.95635713318535]
  Vision-language models (VLMs) are increasingly deployed in high-stakes settings, where a response that is reasonable in general may still be unsafe for a particular user.<n>We study this problem of personalized safety in multimodal systems and introduce MPS-Bench, a benchmark of 5,181 scenarios from 584 real-world images.<n>We find that they almost always respond directly (86-99%) rather than seek missing context, and none exceeds 2.6/5 on personalized safety.<n>Motivated by this mechanism, we propose PRISM, a lightweight input monitor that uses bidirectional cross-modal modulation to predict when a query is likely to require
  arXiv  Detail & Related papers  (2026-09-03T03:58:56Z)
• CareGraph: An Auditable Hybrid AI Framework for Evidence-Grounded Personalized Longitudinal Health Intelligence [0.0]
  CareGraph organizes evidence without diagnosing, predicting outcomes, selecting treatment, or making autonomous clinical decisions.<n>Tests used synthetic cohorts of 400 patients each for development, validation, and holdout.<n>CareGraph offers a safety bounded foundation for intelligent personalized health systems.
  arXiv  Detail & Related papers  (2026-08-25T20:03:52Z)
• AI-generated Images Challenge Visual Trust in High-risk Scenarios [79.40239296310791]
  We introduce SafeIMG, a safety-oriented benchmark spanning 12 public- and individual-safety scenarios generated using GPT Image 2.<n>SafeIMG evaluates whether detectors recognise synthetic images, but also whether their decisions reflect human-identified anomalies.<n>We evaluate specialized synthetic-image detectors and vision-language models (VLMs), and find that neither provides reliable detection.
  arXiv  Detail & Related papers  (2026-07-23T06:56:49Z)
• Measuring and Mitigating Toxicity in Large Language Models: A Comprehensive Replication Study [0.0]
  Large Language Models (LLMs) trained on web-scale corpora inherently absorb toxic patterns from their training data.<n>Decoding-time Experts (DExperts) is an inference-time mitigation technique that steers generation without requiring model retraining.<n>This study contributes to the growing body of work on AI safety by highlighting the gap between explicit and implicit toxicity mitigation.
  arXiv  Detail & Related papers  (2026-05-13T20:12:30Z)
• Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context [82.32380418146656]
  Health-ORSC-Bench is the first large-scale benchmark designed to measure textbfOver-Refusal and textbfSafe Completion quality in healthcare.<n>Our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity.<n>Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants.
  arXiv  Detail & Related papers  (2026-01-25T01:28:52Z)
• I Detect What I Don't Know: Incremental Anomaly Learning with Stochastic Weight Averaging-Gaussian for Oracle-Free Medical Imaging [2.384534878752428]
  We introduce an unsupervised, oracle-free framework that incrementally expands a trusted set of normal samples without any anomaly labels.<n>A frozen pretrained vision backbone is augmented with tiny convolutional adapters, ensuring rapid domain adaptation with negligible computational overhead.<n>On COVID-CXR, ROC-AUC improves from 0.9489 to 0.9982; on Pneumonia CXR, ROC-AUC rises from 0.6834 to 0.8968; and on Brain MRI ND-5, ROC-AUC increases from 0.6041 to 0.7269.
  arXiv  Detail & Related papers  (2025-11-05T23:28:14Z)
• Rethinking Toxicity Evaluation in Large Language Models: A Multi-Label Perspective [104.09817371557476]
  Large language models (LLMs) have achieved impressive results across a range of natural language processing tasks.<n>Their potential to generate harmful content has raised serious safety concerns.<n>We introduce three novel multi-label benchmarks for toxicity detection.
  arXiv  Detail & Related papers  (2025-10-16T06:50:33Z)
• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models [59.45605332033458]
  Safety mechanisms can backfire, causing over-refusal, where models decline benign requests out of excessive caution.<n>No existing benchmark has systematically addressed over-refusal in the visual modality.<n>This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content.
  arXiv  Detail & Related papers  (2025-10-12T23:21:34Z)
• OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models [91.55634905861827]
  Over-refusal is a phenomenon known as $textitover-refusal$ that reduces the practical utility of T2I models.<n>We present OVERT ($textbfOVE$r-$textbfR$efusal evaluation on $textbfT$ext-to-image models), the first large-scale benchmark for assessing over-refusal behaviors.
  arXiv  Detail & Related papers  (2025-05-27T15:42:46Z)
• ArtiFact: A Large-Scale Dataset with Artificial and Factual Images for Generalizable and Robust Synthetic Image Detection [0.3779860024918729]
  This paper assesses the generalizability and robustness of synthetic image detectors in the face of real-world impairments. A proposed multi-class classification scheme, combined with a filter stride reduction strategy addresses social platform impairments. The solution significantly outperforms other top teams by 8.34% on Test 1, 1.26% on Test 2, and 15.08% on Test 3 in the IEEE VIP Cup challenge at ICIP 2022, as measured by the accuracy metric.
  arXiv  Detail & Related papers  (2023-02-23T12:40:36Z)
• Cascading Neural Network Methodology for Artificial Intelligence-Assisted Radiographic Detection and Classification of Lead-Less Implanted Electronic Devices within the Chest [0.7874708385247353]
  This work focused on developing CXR interpretation-assisting Artificial Intelligence (AI) methodology with: 1. 100% detection for LLIED presence/location; and 2. High classification in LLIED typing. For developing the cascading neural network (detection via Faster R-CNN and classification via Inception V3), "ground-truth" CXR annotation (ROI labeling per LLIED), as well as inference display (as Generated Bounding Boxes (GBBs))
  arXiv  Detail & Related papers  (2021-08-25T19:29:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.