Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA

• URL: http://arxiv.org/abs/2505.06356v1
• Date: Fri, 09 May 2025 18:01:50 GMT
• Title: Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA
• Authors: Karthik Reddy Kanjula, Surya Guthikonda, Nahid Alam, Shayekh Bin Islam,
• Abstract summary: This dataset removes 7,531 of toxic image-text pairs in the LLaVA pre-training dataset.<n>We offer guidelines for implementing robust toxicity detection pipelines.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretraining datasets are foundational to the development of multimodal models, yet they often have inherent biases and toxic content from the web-scale corpora they are sourced from. In this paper, we investigate the prevalence of toxicity in LLaVA image-text pretraining dataset, examining how harmful content manifests in different modalities. We present a comprehensive analysis of common toxicity categories and propose targeted mitigation strategies, resulting in the creation of a refined toxicity-mitigated dataset. This dataset removes 7,531 of toxic image-text pairs in the LLaVA pre-training dataset. We offer guidelines for implementing robust toxicity detection pipelines. Our findings underscore the need to actively identify and filter toxic content - such as hate speech, explicit imagery, and targeted harassment - to build more responsible and equitable multimodal systems. The toxicity-mitigated dataset is open source and is available for further research.

Related papers

• GloSS over Toxicity: Understanding and Mitigating Toxicity in LLMs via Global Toxic Subspace [62.68664365246247]
  This paper investigates the underlying mechanisms of toxicity generation in Large Language Models (LLMs)<n>We propose GloSS (Global Toxic Subspace Suppression), a lightweight, four-stage method that mitigates toxicity by identifying and removing the global toxic subspace from the parameters of FFN.
  arXiv  Detail & Related papers  (2025-05-20T08:29:11Z)
• ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs [72.8646625127485]
  Multimodal implicit toxicity appears not only as formal statements in social platforms but also prompts that can lead to toxic dialogs.<n>Despite the success in unimodal text or image moderation, toxicity detection for multimodal content, particularly the multimodal implicit toxicity, remains underexplored.<n>To advance the detection of multimodal implicit toxicity, we build ShieldVLM, a model which identifies implicit toxicity in multimodal statements, prompts and dialogs via deliberative cross-modal reasoning.
  arXiv  Detail & Related papers  (2025-05-20T07:31:17Z)
• Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs [1.7451266777840306]
  Large language models (LLMs) have become integral to various real-world applications, leveraging massive, web-sourced datasets like Common Crawl, C4, and FineWeb for pretraining.<n>Training LLMs on unfiltered data risks perpetuating toxic behaviors, spreading misinformation, and amplifying societal biases.<n>This paper presents a large-scale analysis of inappropriate content across these datasets, offering a comprehensive taxonomy that categorizes harmful webpages into Topical and Toxic based on their intent.
  arXiv  Detail & Related papers  (2025-05-04T06:37:20Z)
• Aligned Probing: Relating Toxic Behavior and Model Internals [66.49887503194101]
  We introduce aligned probing, a novel interpretability framework that aligns the behavior of language models (LMs)<n>Using this framework, we examine over 20 OLMo, Llama, and Mistral models, bridging behavioral and internal perspectives for toxicity for the first time.<n>Our results show that LMs strongly encode information about the toxicity level of inputs and subsequent outputs, particularly in lower layers.
  arXiv  Detail & Related papers  (2025-03-17T17:23:50Z)
• ToxiLab: How Well Do Open-Source LLMs Generate Synthetic Toxicity Data? [29.23490658406256]
  This study explores the potential of open-source LLMs for harmful data synthesis.<n>We evaluate their ability to generate diverse, high-quality harmful data while minimizing hallucination and duplication.<n>Our findings demonstrate that fine-tuned open source LLMs provide scalable and cost-effective solutions to augment toxic content detection datasets.
  arXiv  Detail & Related papers  (2024-11-18T00:21:14Z)
• Unlearnable Examples Detection via Iterative Filtering [84.59070204221366]
  Deep neural networks are proven to be vulnerable to data poisoning attacks. It is quite beneficial and challenging to detect poisoned samples from a mixed dataset. We propose an Iterative Filtering approach for UEs identification.
  arXiv  Detail & Related papers  (2024-08-15T13:26:13Z)
• Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric [16.423707276483178]
  We introduce a robust metric grounded on Large Language Models (LLMs) to flexibly measure toxicity according to the given definition. Our results demonstrate outstanding performance in measuring toxicity within verified factors, improving on conventional metrics by 12 points in the F1 score.
  arXiv  Detail & Related papers  (2024-02-10T07:55:27Z)
• VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models [46.72546879204724]
  In the real-world, datasets may contain dirty samples, such as poisoned samples from backdoor attack, noisy labels in crowdsourcing, and even hybrids of them. Existing detectors only focus on detecting poisoned samples or noisy labels, that are often prone to weak generalization when dealing with dirty samples from other domains. We propose versatile data cleanser (VDC) leveraging the surpassing capabilities of multimodal large language models (MLLM) in cross-modal alignment and reasoning.
  arXiv  Detail & Related papers  (2023-09-28T07:37:18Z)
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity [84.6421260559093]
  This study is the largest set of experiments to validate, quantify, and expose undocumented intuitions about text pretraining. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data.
  arXiv  Detail & Related papers  (2023-05-22T15:57:53Z)
• Facilitating Fine-grained Detection of Chinese Toxic Language: Hierarchical Taxonomy, Resources, and Benchmarks [18.44630180661091]
  Existing datasets lack fine-grained annotation of toxic types and expressions. It is crucial to introduce lexical knowledge to detect the toxicity of posts. In this paper, we facilitate the fine-grained detection of Chinese toxic language.
  arXiv  Detail & Related papers  (2023-05-08T03:50:38Z)
• Toxicity Detection can be Sensitive to the Conversational Context [64.28043776806213]
  We construct and publicly release a dataset of 10,000 posts with two kinds of toxicity labels. We introduce a new task, context sensitivity estimation, which aims to identify posts whose perceived toxicity changes if the context is also considered.
  arXiv  Detail & Related papers  (2021-11-19T13:57:26Z)
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models [93.151822563361]
  Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
  arXiv  Detail & Related papers  (2020-09-24T03:17:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.