SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation

• URL: http://arxiv.org/abs/2606.01481v1
• Date: Sun, 31 May 2026 22:46:35 GMT
• Title: SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation
• Abstract summary: We introduce SafeGen-Bench, a benchmark specifically designed to evaluate the safety of conditional T2V models.<n>Our benchmark defines 10 malicious categories, concentrating on risks related to both temporal sequences and depicted behaviors.<n>We evaluate a variety of conditional T2V models on SafeGen-Bench, and the results indicate that current models struggle to consistently avoid generating malicious content.
• Score: 46.016571758908015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid advancements in text-to-image diffusion models, generative video models (T2V models) like Sora can now produce short synthetic videos from a text prompt or an initial image. However, synthetic video generation -- especially when guided by an initial image -- often poses risks, including the potential creation of illegal, politically sensitive, or unethical content. Existing benchmarks have started to consider the safety of generated videos, but they primarily focus on testing models with malicious text prompts, ignoring the scenario where text prompt and image combination may still lead to harmful video content. In practice, this is a common and challenging issue: videos generated from safe text and image inputs can nonetheless convey harmful information. To bridge this gap, we introduce SafeGen-Bench, a benchmark specifically designed to evaluate the safety of conditional T2V models. Our benchmark defines 10 malicious categories, concentrating on risks related to both temporal sequences and depicted behaviors. SafeGen-Bench consists of carefully selected start frames from diverse image and video sources, paired with corresponding text prompts to simulate realistic inputs. We evaluate a variety of conditional T2V models on SafeGen-Bench, and the results indicate that current models struggle to consistently avoid generating malicious content with unsafety scores reaching up to 44.5, especially under conditions requiring high quality. Furthermore, we assess the effectiveness of both text-based and image-based guardrails on our benchmark, finding that unimodal guardrails alone were insufficient to provide a robust defense, with an 80\% failure rate across seven malicious categories. We hope that SafeGen-Bench will foster the development of safer and more controllable conditional T2V models.

Related papers

• The Price of Consistency: Exploiting Visual Anchors for Multimodal Jailbreaking in Video Generation [83.99528259002504]
  We reveal the visual anchoring effect: by enforcing consistency, the mechanism prevents the generated content from drifting away from the original harmful intent.<n>We propose Decoupling Intent via Visual Anchors (DIVA), a training-free multimodal jailbreak framework for video generation that exploits this vulnerability.<n>To facilitate future research, we additionally contribute TI2SafetyBench, the first safety benchmark for multi-conditional video generation.
  arXiv  Detail & Related papers  (2026-09-07T08:34:00Z)
• VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks [27.060981309996492]
  We present the first systematic benchmark for evaluating video generation safety under vision-centric prompt attacks.<n>We propose VPA-Guard, a retrieval-augmented and self-evolving defense framework.
  arXiv  Detail & Related papers  (2026-06-24T09:00:08Z)
• VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models [57.128876964730644]
  Image-to-Video (I2V) generation models, which condition video generation on reference images, have shown emerging visual instruction-following capability.<n>We propose Visual Instruction Injection (VII), a training-free and transferable jailbreaking framework that disguises the malicious intent of unsafe text prompts as benign visual instructions in the safe reference image.<n>VII achieves Attack Success Rates of up to 83.5% while reducing Refusal Rates to near zero, significantly outperforming existing baselines.
  arXiv  Detail & Related papers  (2026-02-24T15:20:01Z)
• ConceptGuard: Proactive Safety in Text-and-Image-to-Video Generation through Multimodal Risk Detection [27.47621607462884]
  ConceptGuard is a framework for proactively detecting and mitigating unsafe semantics in multimodal video generation.<n>A contrastive detection module identifies latent safety risks by projecting fused image-text inputs into a structured concept space.<n>A semantic suppression mechanism steers the generative process away from unsafe concepts by intervening in the prompt's multimodal conditioning.
  arXiv  Detail & Related papers  (2025-11-24T05:27:05Z)
• VMDT: Decoding the Trustworthiness of Video Foundation Models [77.90980744982079]
  We introduce VMDT, the first unified platform for evaluating text-to-video (T2V) and video-to-text (V2T) models.<n>Through our evaluation of 7 T2V models and 19 V2T models using VMDT, we uncover several significant insights.
  arXiv  Detail & Related papers  (2025-11-07T19:56:00Z)
• SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models [74.11062256255387]
  Text-to-image models are highly vulnerable to adversarial prompts, which can bypass safety measures and produce harmful content.<n>We introduce SafeGuider, a two-step framework designed for robust safety control without compromising generation quality.<n>SafeGuider demonstrates exceptional effectiveness in minimizing attack success rates, achieving a maximum rate of only 5.48% across various attack scenarios.
  arXiv  Detail & Related papers  (2025-10-05T10:24:48Z)
• GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models [65.91565607573786]
  Text-to-image (T2I) models can be misused to generate harmful content, including nudity or violence.<n>Recent research on red-teaming and adversarial attacks against T2I models has notable limitations.<n>We propose GenBreak, a framework that fine-tunes a red-team large language model (LLM) to systematically explore underlying vulnerabilities.
  arXiv  Detail & Related papers  (2025-06-11T09:09:12Z)
• Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs [45.265397990158846]
  Video-SafetyBench is the first benchmark designed to evaluate the safety of LVLMs under video-text attacks.<n>It comprises 2,264 video-text pairs spanning 48 fine-grained unsafe categories.<n>To generate semantically accurate videos for safety evaluation, we design a controllable pipeline that decomposes video semantics into subject images and motion text.
  arXiv  Detail & Related papers  (2025-05-17T05:06:38Z)
• T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks [67.91652526657599]
  We formalize the T2V jailbreak attack as a discrete optimization problem and propose a joint objective-based optimization framework, called T2V-OptJail.<n>We conduct large-scale experiments on several T2V models, covering both open-source models and real commercial closed-source models.<n>The proposed method improves 11.4% and 10.0% over the existing state-of-the-art method in terms of attack success rate.
  arXiv  Detail & Related papers  (2025-05-10T16:04:52Z)
• Safe Text-to-Image Generation: Simply Sanitize the Prompt Embedding [16.188657772178747]
  We propose Embedding Sanitizer (ES), which enhances the safety of text-to-image models by sanitizing inappropriate concepts in prompt embeddings.<n>ES is the first interpretable safe generation framework that assigns a score to each token in the prompt to indicate its potential harmfulness.
  arXiv  Detail & Related papers  (2024-11-15T16:29:02Z)
• T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models [39.15695612766001]
  We introduce T2VSafetyBench, a new benchmark for safety-critical assessments of text-to-video models. We define 12 critical aspects of video generation safety and construct a malicious prompt dataset. No single model excels in all aspects, with different models showing various strengths. There is a trade-off between the usability and safety of text-to-video generative models.
  arXiv  Detail & Related papers  (2024-07-08T14:04:58Z)
• SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models [28.23494821842336]
  Text-to-image models may be tricked into generating not-safe-for-work (NSFW) content. We present SafeGen, a framework to mitigate sexual content generation by text-to-image models.
  arXiv  Detail & Related papers  (2024-04-10T00:26:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.