Towards Understanding Unsafe Video Generation

• URL: http://arxiv.org/abs/2407.12581v1
• Date: Wed, 17 Jul 2024 14:07:22 GMT
• Title: Towards Understanding Unsafe Video Generation
• Authors: Yan Pang, Aiping Xiong, Yang Zhang, Tianhao Wang,
• Abstract summary: Video generation models (VGMs) have demonstrated the capability to synthesize high-quality output. We identify 5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic, Violent/Bloody, and Political. We then study possible defense mechanisms to prevent the generation of unsafe videos.
• Score: 10.269782780518428
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video generation models (VGMs) have demonstrated the capability to synthesize high-quality output. It is important to understand their potential to produce unsafe content, such as violent or terrifying videos. In this work, we provide a comprehensive understanding of unsafe video generation. First, to confirm the possibility that these models could indeed generate unsafe videos, we choose unsafe content generation prompts collected from 4chan and Lexica, and three open-source SOTA VGMs to generate unsafe videos. After filtering out duplicates and poorly generated content, we created an initial set of 2112 unsafe videos from an original pool of 5607 videos. Through clustering and thematic coding analysis of these generated videos, we identify 5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic, Violent/Bloody, and Political. With IRB approval, we then recruit online participants to help label the generated videos. Based on the annotations submitted by 403 participants, we identified 937 unsafe videos from the initial video set. With the labeled information and the corresponding prompts, we created the first dataset of unsafe videos generated by VGMs. We then study possible defense mechanisms to prevent the generation of unsafe videos. Existing defense methods in image generation focus on filtering either input prompt or output results. We propose a new approach called Latent Variable Defense (LVD), which works within the model's internal sampling process. LVD can achieve 0.90 defense accuracy while reducing time and computing resources by 10x when sampling a large number of unsafe prompts.

Related papers

• Loong: Generating Minute-level Long Videos with Autoregressive Language Models [76.59124981781602]
  We propose Loong, a new autoregressive large language models (LLMs)-based video generator that can generate minute-long videos. Specifically, we model the text tokens and video tokens as a unified sequence for autoregressive LLMs and train the model from scratch. Our proposed Loong can be trained on 10-second videos and be extended to generate minute-level long videos conditioned on text prompts.
  arXiv  Detail & Related papers  (2024-10-03T17:59:02Z)
• T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models [39.15695612766001]
  We introduce T2VSafetyBench, a new benchmark for safety-critical assessments of text-to-video models. We define 12 critical aspects of video generation safety and construct a malicious prompt dataset. No single model excels in all aspects, with different models showing various strengths. There is a trade-off between the usability and safety of text-to-video generative models.
  arXiv  Detail & Related papers  (2024-07-08T14:04:58Z)
• DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmark [38.604684882464944]
  We introduce the first AI-generated video detection dataset, GenVideo. It features a large volume of videos, including over one million AI-generated and real videos collected. We introduce a plug-and-play module, named Detail Mamba, to enhance detectors by identifying AI-generated videos.
  arXiv  Detail & Related papers  (2024-05-30T05:36:12Z)
• Unveiling the Potential: Harnessing Deep Metric Learning to Circumvent Video Streaming Encryption [4.522744286037825]
  We propose a deep metric learning framework based on the triplet loss method for encrypted video stream detection. We show that our method scales well to a dataset of over 1000 videos. Our results suggest that this side-channel attack is more broadly applicable than originally thought.
  arXiv  Detail & Related papers  (2024-05-16T08:49:05Z)
• VGMShield: Mitigating Misuse of Video Generative Models [7.963591895964269]
  We introduce VGMShield: a set of three straightforward but pioneering mitigations through the lifecycle of fake video generation. We first try to understand whether there is uniqueness in generated videos and whether we can differentiate them from real videos. Then, we investigate the textittracing problem, which maps a fake video back to a model that generates it.
  arXiv  Detail & Related papers  (2024-02-20T16:39:23Z)
• A Recipe for Scaling up Text-to-Video Generation with Text-free Videos [72.59262815400928]
  Diffusion-based text-to-video generation has witnessed impressive progress in the past year yet still falls behind text-to-image generation. We come up with a novel text-to-video generation framework, termed TF-T2V, which can directly learn with text-free videos.
  arXiv  Detail & Related papers  (2023-12-25T16:37:39Z)
• Malicious or Benign? Towards Effective Content Moderation for Children's Videos [1.0323063834827415]
  This paper introduces our toolkit Malicious or Benign for promoting research on automated content moderation of children's videos. We present 1) a customizable annotation tool for videos, 2) a new dataset with difficult to detect test cases of malicious content, and 3) a benchmark suite of state-of-the-art video classification models.
  arXiv  Detail & Related papers  (2023-05-24T20:33:38Z)
• Video Generation Beyond a Single Clip [76.5306434379088]
  Video generation models can only generate video clips that are relatively short compared with the length of real videos. To generate long videos covering diverse content and multiple events, we propose to use additional guidance to control the video generation process. The proposed approach is complementary to existing efforts on video generation, which focus on generating realistic video within a fixed time window.
  arXiv  Detail & Related papers  (2023-04-15T06:17:30Z)
• Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer [66.56167074658697]
  We present a method that builds on 3D-VQGAN and transformers to generate videos with thousands of frames. Our evaluation shows that our model trained on 16-frame video clips can generate diverse, coherent, and high-quality long videos. We also showcase conditional extensions of our approach for generating meaningful long videos by incorporating temporal information with text and audio.
  arXiv  Detail & Related papers  (2022-04-07T17:59:02Z)
• VPN: Video Provenance Network for Robust Content Attribution [72.12494245048504]
  We present VPN - a content attribution method for recovering provenance information from videos shared online. We learn a robust search embedding for matching such video, using full-length or truncated video queries. Once matched against a trusted database of video clips, associated information on the provenance of the clip is presented to the user.
  arXiv  Detail & Related papers  (2021-09-21T09:07:05Z)
• Playable Video Generation [47.531594626822155]
  We aim at allowing a user to control the generated video by selecting a discrete action at every time step as when playing a video game. The difficulty of the task lies both in learning semantically consistent actions and in generating realistic videos conditioned on the user input. We propose a novel framework for PVG that is trained in a self-supervised manner on a large dataset of unlabelled videos.
  arXiv  Detail & Related papers  (2021-01-28T18:55:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.