T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models

• URL: http://arxiv.org/abs/2407.05965v3
• Date: Sun, 8 Sep 2024 16:19:53 GMT
• Title: T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models
• Authors: Yibo Miao, Yifan Zhu, Yinpeng Dong, Lijia Yu, Jun Zhu, Xiao-Shan Gao,
• Abstract summary: We introduce T2VSafetyBench, a new benchmark for safety-critical assessments of text-to-video models. We define 12 critical aspects of video generation safety and construct a malicious prompt dataset. No single model excels in all aspects, with different models showing various strengths. There is a trade-off between the usability and safety of text-to-video generative models.
• Score: 39.15695612766001
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent development of Sora leads to a new era in text-to-video (T2V) generation. Along with this comes the rising concern about its security risks. The generated videos may contain illegal or unethical content, and there is a lack of comprehensive quantitative understanding of their safety, posing a challenge to their reliability and practical deployment. Previous evaluations primarily focus on the quality of video generation. While some evaluations of text-to-image models have considered safety, they cover fewer aspects and do not address the unique temporal risk inherent in video generation. To bridge this research gap, we introduce T2VSafetyBench, a new benchmark designed for conducting safety-critical assessments of text-to-video models. We define 12 critical aspects of video generation safety and construct a malicious prompt dataset including real-world prompts, LLM-generated prompts and jailbreak attack-based prompts. Based on our evaluation results, we draw several important findings, including: 1) no single model excels in all aspects, with different models showing various strengths; 2) the correlation between GPT-4 assessments and manual reviews is generally high; 3) there is a trade-off between the usability and safety of text-to-video generative models. This indicates that as the field of video generation rapidly advances, safety risks are set to surge, highlighting the urgency of prioritizing video safety. We hope that T2VSafetyBench can provide insights for better understanding the safety of video generation in the era of generative AI.

Related papers

• BadVideo: Stealthy Backdoor Attack against Text-to-Video Generation [37.055665794706336]
  Text-to-video (T2V) generative models have rapidly advanced and found widespread applications across fields like entertainment, education, and marketing. We observe that in T2V generation tasks, the generated videos often contain substantial redundant information not explicitly specified in the text prompts. We introduce BadVideo, the first backdoor attack framework tailored for T2V generation.
  arXiv  Detail & Related papers  (2025-04-23T17:34:48Z)
• T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models [88.63040835652902]
  Text to video models are vulnerable to jailbreak attacks, where specially crafted prompts bypass safety mechanisms and lead to the generation of harmful or unsafe content. We propose T2VShield, a comprehensive and model agnostic defense framework designed to protect text to video models from jailbreak threats. Our method systematically analyzes the input, model, and output stages to identify the limitations of existing defenses.
  arXiv  Detail & Related papers  (2025-04-22T01:18:42Z)
• VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness [76.16523963623537]
  We introduce VBench-2.0, a benchmark designed to evaluate video generative models for intrinsic faithfulness. VBench-2.0 assesses five key dimensions: Human Fidelity, Controllability, Creativity, Physics, and Commonsense. By pushing beyond superficial faithfulness toward intrinsic faithfulness, VBench-2.0 aims to set a new standard for the next generation of video generative models.
  arXiv  Detail & Related papers  (2025-03-27T17:57:01Z)
• VPO: Aligning Text-to-Video Generation Models with Prompt Optimization [80.86205966195593]
  Video generation models are typically trained on text-to-video pairs with highly detailed and carefully crafted descriptions. We introduce VPO, a principled framework that optimize prompts based on three core principles: harmlessness, accuracy, and helpfulness. Our experiments demonstrate that VPO significantly improves safety, alignment, and video quality compared to baseline methods.
  arXiv  Detail & Related papers  (2025-03-26T12:28:20Z)
• Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findings [51.65890794988425]
  This study presents the first comprehensive safety evaluation of the DeepSeek models. Our evaluation encompasses DeepSeek's latest generation of large language models, multimodal large language models, and text-to-image models.
  arXiv  Detail & Related papers  (2025-03-19T10:44:37Z)
• T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation [39.45602029655288]
  T2ISafety is a safety benchmark that evaluates T2I models across three key domains: toxicity, fairness, and bias. We build a large-scale T2I dataset with 68K manually annotated images and train an evaluator capable of detecting critical risks. We evaluate 12 prominent diffusion models on T2ISafety and reveal several concerns including persistent issues with racial fairness, a tendency to generate toxic content, and significant variation in privacy protection across the models.
  arXiv  Detail & Related papers  (2025-01-22T03:29:43Z)
• Safe Text-to-Image Generation: Simply Sanitize the Prompt Embedding [13.481343482138888]
  We propose a vision-agnostic safe generation framework, Embedding Sanitizer (ES) ES focuses on erasing inappropriate concepts from prompt embeddings and uses the sanitized embeddings to guide the model for safe generation. ES significantly outperforms existing safeguards in terms of interpretability and controllability while maintaining generation quality.
  arXiv  Detail & Related papers  (2024-11-15T16:29:02Z)
• SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation [65.30207993362595]
  Unlearning/editing-based methods for safe generation remove harmful concepts from models but face several challenges. We propose SAFREE, a training-free approach for safe T2I and T2V. We detect a subspace corresponding to a set of toxic concepts in the text embedding space and steer prompt embeddings away from this subspace.
  arXiv  Detail & Related papers  (2024-10-16T17:32:23Z)
• Multimodal Situational Safety [73.63981779844916]
  We present the first evaluation and analysis of a novel safety challenge termed Multimodal Situational Safety. For an MLLM to respond safely, whether through language or action, it often needs to assess the safety implications of a language query within its corresponding visual context. We develop the Multimodal Situational Safety benchmark (MSSBench) to assess the situational safety performance of current MLLMs.
  arXiv  Detail & Related papers  (2024-10-08T16:16:07Z)
• TrojVLM: Backdoor Attack Against Vision Language Models [50.87239635292717]
  This study introduces TrojVLM, the first exploration of backdoor attacks aimed at Vision Language Models (VLMs) TrojVLM inserts predetermined target text into output text when encountering poisoned images. A novel semantic preserving loss is proposed to ensure the semantic integrity of the original image content.
  arXiv  Detail & Related papers  (2024-09-28T04:37:09Z)
• Towards Understanding Unsafe Video Generation [10.269782780518428]
  Video generation models (VGMs) have demonstrated the capability to synthesize high-quality output. We identify 5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic, Violent/Bloody, and Political. We then study possible defense mechanisms to prevent the generation of unsafe videos.
  arXiv  Detail & Related papers  (2024-07-17T14:07:22Z)
• ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users [18.3621509910395]
  We propose a novel Automatic Red-Teaming framework, ART, to evaluate the safety risks of text-to-image models. With our comprehensive experiments, we reveal the toxicity of the popular open-source text-to-image models. We also introduce three large-scale red-teaming datasets for studying the safety risks associated with text-to-image models.
  arXiv  Detail & Related papers  (2024-05-24T07:44:27Z)
• Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation [19.06501699814924]
  We build the Adversarial Nibbler Challenge, a red-teaming methodology for crowdsourcing implicitly adversarial prompts. The challenge is run in consecutive rounds to enable a sustained discovery and analysis of safety pitfalls in T2I models. We find that 14% of images that humans consider harmful are mislabeled as safe'' by machines.
  arXiv  Detail & Related papers  (2024-02-14T22:21:12Z)
• Towards A Better Metric for Text-to-Video Generation [102.16250512265995]
  Generative models have demonstrated remarkable capability in synthesizing high-quality text, images, and videos. We introduce a novel evaluation pipeline, the Text-to-Video Score (T2VScore) This metric integrates two pivotal criteria: (1) Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and (2) Video Quality, which evaluates the video's overall production caliber with a mixture of experts.
  arXiv  Detail & Related papers  (2024-01-15T15:42:39Z)
• GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation [100.23111948079037]
  GPT4Video is a unified multi-model framework that empowers Large Language Models with the capability of both video understanding and generation. Specifically, we develop an instruction-following-based approach integrated with the stable diffusion generative model, which has demonstrated to effectively and securely handle video generation scenarios.
  arXiv  Detail & Related papers  (2023-11-25T04:05:59Z)
• Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models [6.475537049815622]
  Adversarial Nibbler is a data-centric challenge, part of the DataPerf challenge suite, organized and supported by Kaggle and MLCommons.
  arXiv  Detail & Related papers  (2023-05-22T15:02:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.