On-Device Content Moderation

• URL: http://arxiv.org/abs/2107.11845v1
• Date: Sun, 25 Jul 2021 17:06:01 GMT
• Title: On-Device Content Moderation
• Abstract summary: We present a novel on-device solutionfor detecting NSFW images. We have curated a dataset comprising of three major categories, namely nude, semi-nude and safe images. The model achieves F1 scoreof 0.91 with 95% precision and 88% recall on our customNSFW16k dataset and 0.92 MAP on NPDI dataset.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: With the advent of internet, not safe for work(NSFW) content moderation is a major problem today. Since,smartphones are now part of daily life of billions of people,it becomes even more important to have a solution which coulddetect and suggest user about potential NSFW content present ontheir phone. In this paper we present a novel on-device solutionfor detecting NSFW images. In addition to conventional porno-graphic content moderation, we have also included semi-nudecontent moderation as it is still NSFW in a large demography.We have curated a dataset comprising of three major categories,namely nude, semi-nude and safe images. We have created anensemble of object detector and classifier for filtering of nudeand semi-nude contents. The solution provides unsafe body partannotations along with identification of semi-nude images. Weextensively tested our proposed solution on several public datasetand also on our custom dataset. The model achieves F1 scoreof 0.91 with 95% precision and 88% recall on our customNSFW16k dataset and 0.92 MAP on NPDI dataset. Moreover itachieves average 0.002 false positive rate on a collection of safeimage open datasets.

Related papers

• VModA: An Effective Framework for Adaptive NSFW Image Moderation [36.86383821944732]
  Not Safe/Suitable for Work (NSFW) content is rampant on social networks and poses serious harm to citizens, especially minors.<n>Current detection methods mainly rely on deep learning-based image recognition and classification.<n>We propose VModA, a framework that adapts to diverse moderation rules and handles complex, semantically rich NSFW content.
  arXiv  Detail & Related papers  (2025-05-29T12:11:28Z)
• Towards Safe Synthetic Image Generation On the Web: A Multimodal Robust NSFW Defense and Million Scale Dataset [20.758637391023345]
  A multimodal defense is developed to distinguish safe and NSFW text and images. Our model performs well against existing SOTA NSFW detection methods in terms of accuracy and recall.
  arXiv  Detail & Related papers  (2025-04-16T02:10:42Z)
• Clean Image May be Dangerous: Data Poisoning Attacks Against Deep Hashing [71.30876587855867]
  We show that even clean query images can be dangerous, inducing malicious target retrieval results, like undesired or illegal images. Specifically, we first train a surrogate model to simulate the behavior of the target deep hashing model. Then, a strict gradient matching strategy is proposed to generate the poisoned images.
  arXiv  Detail & Related papers  (2025-03-27T07:54:27Z)
• Training-Free Safe Denoisers for Safe Use of Diffusion Models [49.045799120267915]
  Powerful diffusion models (DMs) are often misused to generate not-safe-for-work (NSFW) content or generate copyrighted material or data of individuals who wish to be forgotten. We develop a practical algorithm that produces high-quality samples while avoiding negation areas of the data distribution. These results hint at the great potential of our training-free safe denoiser for using DMs more safely.
  arXiv  Detail & Related papers  (2025-02-11T23:14:39Z)
• Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in Generated Images [5.150015329535525]
  We identify a novel threat: the generation of NSFW text embedded within images. This includes offensive language, such as insults, racial slurs, and sexually explicit terms. Existing mitigation techniques fail to prevent harmful text generation while substantially degrading text generation. To advance research in this area, we introduce ToxicBench, an open-source benchmark for evaluating NSFW text generation in images.
  arXiv  Detail & Related papers  (2025-02-07T16:39:39Z)
• Detecting and Interpreting NSFW Prompts in Text-to-Image Models through Uncovering Harmful Semantics [48.93115678681624]
  Malicious users exploit text-to-image (T2I) models to generate Not-Safe-for-Work (NSFW) images.<n>We propose HiddenGuard, a framework leveraging the hidden states of T2I models to detect NSFW prompts.<n>Our experiments show that HiddenGuard significantly outperforms both commercial and open-source moderation tools.
  arXiv  Detail & Related papers  (2024-12-24T03:17:45Z)
• Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generation [15.703408347981776]
  We propose an innovative framework named textitBuster, which injects backdoors into the text encoder to prevent NSFW content generation. Buster leverages deep semantic information rather than explicit prompts as triggers, redirecting NSFW prompts towards targeted benign prompts. Our experiments denote that Buster outperforms nine state-of-the-art baselines, achieving a superior NSFW content removal rate of at least 91.2%.
  arXiv  Detail & Related papers  (2024-12-10T07:18:51Z)
• ShieldDiff: Suppressing Sexual Content Generation from Diffusion Models through Reinforcement Learning [7.099258248662009]
  There is a potential risk that text-to-image (T2I) model can generate unsafe images with uncomfortable contents. In our work, we focus on eliminating the NSFW (not safe for work) content generation from T2I model. We propose a customized reward function consisting of the CLIP (Contrastive Language-Image Pre-training) and nudity rewards to prune the nudity contents.
  arXiv  Detail & Related papers  (2024-10-04T19:37:56Z)
• SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models [28.23494821842336]
  Text-to-image models may be tricked into generating not-safe-for-work (NSFW) content. We present SafeGen, a framework to mitigate sexual content generation by text-to-image models.
  arXiv  Detail & Related papers  (2024-04-10T00:26:08Z)
• Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models [42.19184265811366]
  We introduce a novel approach to enhancing the safety of vision-and-language models by diminishing their sensitivity to NSFW (not safe for work) inputs. We show how this can be done by fine-tuning a CLIP model on synthetic data obtained from a large language model trained to convert between safe and unsafe sentences.
  arXiv  Detail & Related papers  (2023-11-27T19:02:17Z)
• Into the LAIONs Den: Investigating Hate in Multimodal Datasets [67.21783778038645]
  This paper investigates the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. We found that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively. We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text.
  arXiv  Detail & Related papers  (2023-11-06T19:00:05Z)
• HOD: A Benchmark Dataset for Harmful Object Detection [3.755082744150185]
  We present a new benchmark dataset for harmful object detection. Our proposed dataset contains more than 10,000 images across 6 categories that might be harmful. We have conducted extensive experiments to evaluate the effectiveness of our proposed dataset.
  arXiv  Detail & Related papers  (2023-10-08T15:00:38Z)
• DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-image Diffusion Models [79.71665540122498]
  We propose a method for detecting unauthorized data usage by planting the injected content into the protected dataset. Specifically, we modify the protected images by adding unique contents on these images using stealthy image warping functions. By analyzing whether the model has memorized the injected content, we can detect models that had illegally utilized the unauthorized data.
  arXiv  Detail & Related papers  (2023-07-06T16:27:39Z)
• Privacy-Preserving Image Acquisition Using Trainable Optical Kernel [50.1239616836174]
  We propose a trainable image acquisition method that removes the sensitive identity revealing information in the optical domain before it reaches the image sensor. As the sensitive content is suppressed before it reaches the image sensor, it does not enter the digital domain therefore is unretrievable by any sort of privacy attack.
  arXiv  Detail & Related papers  (2021-06-28T11:08:14Z)
• A Study of Face Obfuscation in ImageNet [94.2949777826947]
  In this paper, we explore image obfuscation in the ImageNet challenge. Most categories in the ImageNet challenge are not people categories; nevertheless, many incidental people are in the images. We benchmark various deep neural networks on face-blurred images and observe a disparate impact on different categories. Results show that features learned on face-blurred images are equally transferable.
  arXiv  Detail & Related papers  (2021-03-10T17:11:34Z)
• Contemplating real-world object classification [53.10151901863263]
  We reanalyze the ObjectNet dataset recently proposed by Barbu et al. containing objects in daily life situations. We find that applying deep models to the isolated objects, rather than the entire scene as is done in the original paper, results in around 20-30% performance improvement.
  arXiv  Detail & Related papers  (2021-03-08T23:29:59Z)
• Large image datasets: A pyrrhic win for computer vision? [2.627046865670577]
  We investigate problematic practices and consequences of large scale vision datasets. We examine broad issues such as the question of consent and justice as well as specific concerns such as the inclusion of verifiably pornographic images in datasets.
  arXiv  Detail & Related papers  (2020-06-24T06:41:32Z)
• InfoScrub: Towards Attribute Privacy by Targeted Obfuscation [77.49428268918703]
  We study techniques that allow individuals to limit the private information leaked in visual data. We tackle this problem in a novel image obfuscation framework. We find our approach generates obfuscated images faithful to the original input images, and additionally increase uncertainty by 6.2$times$ (or up to 0.85 bits) over the non-obfuscated counterparts.
  arXiv  Detail & Related papers  (2020-05-20T19:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.