Validating Multimedia Content Moderation Software via Semantic Fusion

• URL: http://arxiv.org/abs/2305.13623v1
• Date: Tue, 23 May 2023 02:44:15 GMT
• Title: Validating Multimedia Content Moderation Software via Semantic Fusion
• Authors: Wenxuan Wang, Jingyuan Huang, Chang Chen, Jiazhen Gu, Jianping Zhang, Weibin Wu, Pinjia He, Michael Lyu
• Abstract summary: We introduce Semantic Fusion, a general, effective methodology for validating multimedia content moderation software. We employ DUO to test five commercial content moderation software and two state-of-the-art models against three kinds of toxic content. The results show that DUO achieves up to 100% error finding rate (EFR) when testing moderation software.
• Score: 16.322773343799575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The exponential growth of social media platforms, such as Facebook and TikTok, has revolutionized communication and content publication in human society. Users on these platforms can publish multimedia content that delivers information via the combination of text, audio, images, and video. Meanwhile, the multimedia content release facility has been increasingly exploited to propagate toxic content, such as hate speech, malicious advertisements, and pornography. To this end, content moderation software has been widely deployed on these platforms to detect and blocks toxic content. However, due to the complexity of content moderation models and the difficulty of understanding information across multiple modalities, existing content moderation software can fail to detect toxic content, which often leads to extremely negative impacts. We introduce Semantic Fusion, a general, effective methodology for validating multimedia content moderation software. Our key idea is to fuse two or more existing single-modal inputs (e.g., a textual sentence and an image) into a new input that combines the semantics of its ancestors in a novel manner and has toxic nature by construction. This fused input is then used for validating multimedia content moderation software. We realized Semantic Fusion as DUO, a practical content moderation software testing tool. In our evaluation, we employ DUO to test five commercial content moderation software and two state-of-the-art models against three kinds of toxic content. The results show that DUO achieves up to 100% error finding rate (EFR) when testing moderation software. In addition, we leverage the test cases generated by DUO to retrain the two models we explored, which largely improves model robustness while maintaining the accuracy on the original test set.

Related papers

• ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos [46.148023197749396]
  ToxVidLM incorporates three key modules - the multimodal module, Cross-Modal Synchronization module, and Multitask module. This paper introduces a benchmark dataset consisting of 931 videos with 4021 code-mixed Hindi-English utterances collected from YouTube.
  arXiv  Detail & Related papers  (2024-05-31T05:40:56Z)
• Detecting and Grounding Multi-Modal Media Manipulation and Beyond [93.08116982163804]
  We highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM4) DGM4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content. We propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities.
  arXiv  Detail & Related papers  (2023-09-25T15:05:46Z)
• An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software [64.367830425115]
  Social media platforms are being increasingly misused to spread toxic content, including hate speech, malicious advertising, and pornography. Despite tremendous efforts in developing and deploying content moderation methods, malicious users can evade moderation by embedding texts into images. We propose a metamorphic testing framework for content moderation software.
  arXiv  Detail & Related papers  (2023-08-18T20:33:06Z)
• Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection [53.48346699224921]
  A novel Knowledge-guided Dualconsistency Network is proposed to detect rumors with multimedia contents. It uses two consistency detectionworks to capture the inconsistency at the cross-modal level and the content-knowledge level simultaneously. It also enables robust multi-modal representation learning under different missing visual modality conditions.
  arXiv  Detail & Related papers  (2023-06-03T15:32:20Z)
• Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion [21.042970740577648]
  We present a Multi-grained Multi-modal Fusion Network (MMFN) for fake news detection. Inspired by the multi-grained process of human assessment of news authenticity, we respectively employ two Transformer-based pre-trained models to encode token-level features from text and images. The multi-modal module fuses fine-grained features, taking into account coarse-grained features encoded by the CLIP encoder.
  arXiv  Detail & Related papers  (2023-04-03T09:13:59Z)
• MTTM: Metamorphic Testing for Textual Content Moderation Software [11.759353169546646]
  Social media platforms have been increasingly exploited to propagate toxic content. malicious users can evade moderation by changing only a few words in the toxic content. We propose MTTM, a Metamorphic Testing framework for Textual content Moderation software.
  arXiv  Detail & Related papers  (2023-02-11T14:44:39Z)
• Countering Malicious Content Moderation Evasion in Online Social Networks: Simulation and Detection of Word Camouflage [64.78260098263489]
  Twisting and camouflaging keywords are among the most used techniques to evade platform content moderation systems. This article contributes significantly to countering malicious information by developing multilingual tools to simulate and detect new methods of evasion of content.
  arXiv  Detail & Related papers  (2022-12-27T16:08:49Z)
• GAME-ON: Graph Attention Network based Multimodal Fusion for Fake News Detection [6.037721620350107]
  We propose GAME-ON, a Graph Neural Network based end-to-end trainable framework to learn more robust data representations for multimodal fake news detection. Our model outperforms on Twitter by an average of 11% and keeps competitive performance on Weibo, within a 2.6% margin, while using 65% fewer parameters than the best comparable state-of-the-art baseline.
  arXiv  Detail & Related papers  (2022-02-25T03:27:37Z)
• VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles [63.32111010686954]
  We propose the task of Video-based Multimodal Summarization with Multimodal Output (VMSMO) The main challenge in this task is to jointly model the temporal dependency of video with semantic meaning of article. We propose a Dual-Interaction-based Multimodal Summarizer (DIMS), consisting of a dual interaction module and multimodal generator.
  arXiv  Detail & Related papers  (2020-10-12T02:19:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.