Open Models, Open Risks: Measuring Unsafe Generation in Text-to-Image Models In the Wild

• URL: http://arxiv.org/abs/2607.07827v1
• Date: Wed, 08 Jul 2026 18:07:35 GMT
• Title: Open Models, Open Risks: Measuring Unsafe Generation in Text-to-Image Models In the Wild
• Abstract summary: Existing safety studies on text-to-image (T2I) jailbreaks are largely conducted in controlled in-the-lab settings.<n>We present a large-scale empirical study of in-the-wild T2I safety through the lens of jailbreak.
• Score: 12.266069266190259
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Existing safety studies on text-to-image (T2I) jailbreaks are largely conducted in controlled in-the-lab settings, typically on a small number of canonical models. As a result, the current safety status of the rapidly growing in-the-wild T2I ecosystem remains unclear. This uncertainty is amplified by two factors: existing detector-based metrics are designed for controlled evaluation, and in-the-wild risks may arise not only from adversarial prompting, but also from unsafe release practices and unsafe model derivatives. In this paper, we present a large-scale empirical study of in-the-wild T2I safety through the lens of jailbreak. We first show that detector-only jailbreak metrics substantially overestimate practical risk over in the wild due to semantic drift and generation artifacts, and we introduce Advanced ASR to better capture semantically valid and visually plausible unsafe generation. Using this refined metric, we evaluate 200+ in-the-wild T2I models from Hugging Face under three representative jailbreak attacks. Our results show that many downstream models retain a non-trivial degree of safety even without explicit post-hoc safeguards, indicating that safety degradation in the wild is neither universal nor uniform. At the same time, we identify a set of high-risk models, including explicitly NSFW-oriented releases as well as seemingly benign models whose unsafe behavior is only exposed through systematic evaluation. We further trace these models to their release context and report high-risk cases to Hugging Face.

Related papers

• Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack [19.201635180438313]
  Large language models (LLMs) are rigorously aligned to refuse harmful requests.<n>In this work, we reveal that this advanced safety awareness inadvertently introduces a fatal vulnerability.
  arXiv  Detail & Related papers  (2026-06-04T02:36:41Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• Read the Scene, Not the Script: Outcome-Aware Safety for LLMs [22.814397376238755]
  We trace both to a common cause: current models reason weakly about links between actions and outcomes.<n>We define this failure mode as Consequence-blindness.<n>Models fine-tuned on CS-Chain-4k show clear gains against semantic-camouflage jailbreaks.
  arXiv  Detail & Related papers  (2025-10-05T18:46:49Z)
• SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models [74.11062256255387]
  Text-to-image models are highly vulnerable to adversarial prompts, which can bypass safety measures and produce harmful content.<n>We introduce SafeGuider, a two-step framework designed for robust safety control without compromising generation quality.<n>SafeGuider demonstrates exceptional effectiveness in minimizing attack success rates, achieving a maximum rate of only 5.48% across various attack scenarios.
  arXiv  Detail & Related papers  (2025-10-05T10:24:48Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models [22.796169894587475]
  We propose a general framework that accurately detects unknown jailbreak attacks by shifting the focus from attack-specific learning to task-specific learning.<n>Experiments show that our method consistently higher detection AUROC on diverse unknown attacks while improving efficiency.
  arXiv  Detail & Related papers  (2025-08-08T16:13:28Z)
• GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models [65.91565607573786]
  Text-to-image (T2I) models can be misused to generate harmful content, including nudity or violence.<n>Recent research on red-teaming and adversarial attacks against T2I models has notable limitations.<n>We propose GenBreak, a framework that fine-tunes a red-team large language model (LLM) to systematically explore underlying vulnerabilities.
  arXiv  Detail & Related papers  (2025-06-11T09:09:12Z)
• OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models [91.55634905861827]
  Over-refusal is a phenomenon known as $textitover-refusal$ that reduces the practical utility of T2I models.<n>We present OVERT ($textbfOVE$r-$textbfR$efusal evaluation on $textbfT$ext-to-image models), the first large-scale benchmark for assessing over-refusal behaviors.
  arXiv  Detail & Related papers  (2025-05-27T15:42:46Z)
• SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning [76.56522719330911]
  Large Reasoning Models (LRMs) introduce a new generation paradigm of explicitly reasoning before answering.<n>LRMs pose great safety risks against harmful queries and adversarial attacks.<n>We propose SafeKey to better activate the safety aha moment in the key sentence.
  arXiv  Detail & Related papers  (2025-05-22T03:46:03Z)
• T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation [39.45602029655288]
  T2ISafety is a safety benchmark that evaluates T2I models across three key domains: toxicity, fairness, and bias.<n>We build a large-scale T2I dataset with 68K manually annotated images and train an evaluator capable of detecting critical risks.<n>We evaluate 12 prominent diffusion models on T2ISafety and reveal several concerns including persistent issues with racial fairness, a tendency to generate toxic content, and significant variation in privacy protection across the models.
  arXiv  Detail & Related papers  (2025-01-22T03:29:43Z)
• Multimodal Pragmatic Jailbreak on Text-to-image Models [42.53834190690214]
  This work introduces a novel type of jailbreak, which triggers T2I models to generate the image with visual text.<n>We benchmark nine representative T2I models, including two closed-source commercial models.<n>Our work provides a foundation for further development towards more secure and reliable T2I models.
  arXiv  Detail & Related papers  (2024-09-27T21:23:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.