Toxicity in the Decentralized Web and the Potential for Model Sharing

• URL: http://arxiv.org/abs/2204.12709v1
• Date: Wed, 27 Apr 2022 05:42:51 GMT
• Title: Toxicity in the Decentralized Web and the Potential for Model Sharing
• Abstract summary: "Decentralised Web" (DW) relies on independent servers that mesh together in a peer-to-peer fashion to deliver a range of services. We quantify the presence of toxic content using a dataset of 9.9M posts from 117K users on Pleroma. We show that automating per-instance content moderation is challenging due to the lack of sufficient training data available and the effort required in labelling.
• Score: 7.499765577591378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The "Decentralised Web" (DW) is an evolving concept, which encompasses technologies aimed at providing greater transparency and openness on the web. The DW relies on independent servers (aka instances) that mesh together in a peer-to-peer fashion to deliver a range of services (e.g. micro-blogs, image sharing, video streaming). However, toxic content moderation in this decentralised context is challenging. This is because there is no central entity that can define toxicity, nor a large central pool of data that can be used to build universal classifiers. It is therefore unsurprising that there have been several high-profile cases of the DW being misused to coordinate and disseminate harmful material. Using a dataset of 9.9M posts from 117K users on Pleroma (a popular DW microblogging service), we quantify the presence of toxic content. We find that toxic content is prevalent and spreads rapidly between instances. We show that automating per-instance content moderation is challenging due to the lack of sufficient training data available and the effort required in labelling. We therefore propose and evaluate ModPair, a model sharing system that effectively detects toxic content, gaining an average per-instance macro-F1 score 0.89.

Related papers

• Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation [58.10046058931267]
  Distributed Implicit Harm (DIH) is where harm arises from relations among components distributed along a decomposition axis of the video.<n>We study two representative cases: temporally distributed harm across visual segments (DIH-T) and cross-modal harm between audio and visual streams (DIH-M)<n>We develop a multi-agent framework that composes individually benign components into harmful scenarios and generates diverse DIH videos with explicit reasoning annotations.
  arXiv  Detail & Related papers  (2026-08-31T18:17:49Z)
• UniNDM: A Unified Noise-driven Detection and Mitigation Framework Against Sexual Content in Text-to-Image Generation [90.0896070825457]
  We propose a unified noise-driven framework that rethinks safety mechanisms through the lens of noise dynamics in diffusion processes.<n>Our key insight is that early-stage predicted noise exhibits inherent separability between normal and sexually explicit content.<n>We develop a lightweight noise-based detector achieving superior accuracy with virtually no computational overhead.
  arXiv  Detail & Related papers  (2026-07-18T14:03:39Z)
• Unveiling Covert Toxicity in Multimodal Data via Toxicity Association Graphs: A Graph-Based Metric and Interpretable Detection Framework [58.01529356381494]
  We propose a novel detection framework based on Toxicity Association Graphs (TAGs)<n>We introduce the first quantifiable metric for hidden toxicity, the Multimodal Toxicity Covertness (MTC)<n>Our approach enables precise identification of covert toxicity while preserving full interpretability of the decision-making process.
  arXiv  Detail & Related papers  (2026-02-03T08:54:25Z)
• Rethinking Toxicity Evaluation in Large Language Models: A Multi-Label Perspective [104.09817371557476]
  Large language models (LLMs) have achieved impressive results across a range of natural language processing tasks.<n>Their potential to generate harmful content has raised serious safety concerns.<n>We introduce three novel multi-label benchmarks for toxicity detection.
  arXiv  Detail & Related papers  (2025-10-16T06:50:33Z)
• Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline [56.790045049514326]
  Two major forms of deception dominate: human-crafted misinformation and AI-generated content.<n>We propose Unified Multimodal Fake Content Detection (UMFDet), a framework designed to handle both forms of deception.<n>UMFDet achieves robust and consistent performance across both misinformation types, outperforming specialized baselines.
  arXiv  Detail & Related papers  (2025-09-30T09:26:32Z)
• ToxicTAGS: Decoding Toxic Memes with Rich Tag Annotations [3.708799808977489]
  We introduce a first-of-its-kind dataset of 6,300 real-world meme-based posts annotated in two stages: (i) binary classification into toxic and normal, and (ii) fine-grained labelling of toxic memes as hateful, dangerous, or offensive.<n>A key feature of this dataset is that it is enriched with auxiliary metadata of socially relevant tags, enhancing the context of each meme.
  arXiv  Detail & Related papers  (2025-08-06T07:46:14Z)
• ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs [72.8646625127485]
  Multimodal implicit toxicity appears not only as formal statements in social platforms but also prompts that can lead to toxic dialogs.<n>Despite the success in unimodal text or image moderation, toxicity detection for multimodal content, particularly the multimodal implicit toxicity, remains underexplored.<n>To advance the detection of multimodal implicit toxicity, we build ShieldVLM, a model which identifies implicit toxicity in multimodal statements, prompts and dialogs via deliberative cross-modal reasoning.
  arXiv  Detail & Related papers  (2025-05-20T07:31:17Z)
• Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA [0.0]
  This dataset removes 7,531 of toxic image-text pairs in the LLaVA pre-training dataset.<n>We offer guidelines for implementing robust toxicity detection pipelines.
  arXiv  Detail & Related papers  (2025-05-09T18:01:50Z)
• Collaborative Content Moderation in the Fediverse [8.001727491022999]
  This paper focuses on one particular challenge: content moderation, e.g. the need to remove spam or hate speech. We design and evaluate FedMod, a collaborative content moderation system based on federated learning. Our system enables servers to exchange parameters of partially trained local content moderation models with similar servers, creating a federated model shared among collaborating servers.
  arXiv  Detail & Related papers  (2025-01-10T11:12:03Z)
• U-GIFT: Uncertainty-Guided Firewall for Toxic Speech in Few-Shot Scenario [13.954929026841413]
  We propose an uncertainty-guided firewall for toxic speech in few-shot scenarios, U-GIFT. U-GIFT combines active learning with Bayesian Neural Networks (BNNs) to automatically identify high-quality samples from unlabeled data. In the 5-shot setting, it achieves a 14.92% performance improvement over the basic model.
  arXiv  Detail & Related papers  (2025-01-01T17:47:22Z)
• Toxicity of the Commons: Curating Open-Source Pre-Training Data [6.137272725645159]
  We propose a data curation pipeline to reduce harmful outputs by models trained on public domain data. Current state-of-the-art approaches to toxicity filtering are often infeasible or inappropriate for open data models. We create a custom training dataset, ToxicCommons, which is composed of texts which have been classified across five different dimensions.
  arXiv  Detail & Related papers  (2024-10-29T23:00:05Z)
• ToVo: Toxicity Taxonomy via Voting [25.22398575368979]
  We propose a dataset creation mechanism that integrates voting and chain-of-thought processes. Our methodology ensures diverse classification metrics for each sample. We utilize the dataset created through our proposed mechanism to train our model.
  arXiv  Detail & Related papers  (2024-06-21T02:35:30Z)
• Decentralised Moderation for Interoperable Social Networks: A Conversation-based Approach for Pleroma and the Fediverse [10.791075186479622]
  We propose a decentralised conversation-aware content moderation approach suitable for the fediverse. Our approach employs a graph deep learning model (GraphNLI) trained locally on each server. Our model effectively detects toxicity on larger instances, exclusively trained using their local post information.
  arXiv  Detail & Related papers  (2024-04-03T20:29:40Z)
• Robust Domain Misinformation Detection via Multi-modal Feature Alignment [49.89164555394584]
  We propose a robust domain and cross-modal approach for multi-modal misinformation detection. It reduces the domain shift by aligning the joint distribution of textual and visual modalities. We also propose a framework that simultaneously considers application scenarios of domain generalization.
  arXiv  Detail & Related papers  (2023-11-24T07:06:16Z)
• Into the LAIONs Den: Investigating Hate in Multimodal Datasets [67.21783778038645]
  This paper investigates the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. We found that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively. We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text.
  arXiv  Detail & Related papers  (2023-11-06T19:00:05Z)
• Validating Multimedia Content Moderation Software via Semantic Fusion [16.322773343799575]
  We introduce Semantic Fusion, a general, effective methodology for validating multimedia content moderation software. We employ DUO to test five commercial content moderation software and two state-of-the-art models against three kinds of toxic content. The results show that DUO achieves up to 100% error finding rate (EFR) when testing moderation software.
  arXiv  Detail & Related papers  (2023-05-23T02:44:15Z)
• Poisoning Web-Scale Training Datasets is Practical [73.34964403079775]
  We introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model's performance. First attack, split-view poisoning, exploits the mutable nature of internet content to ensure a dataset annotator's initial view of the dataset differs from the view downloaded by subsequent clients. Second attack, frontrunning poisoning, targets web-scale datasets that periodically snapshot crowd-sourced content.
  arXiv  Detail & Related papers  (2023-02-20T18:30:54Z)
• Toxicity Detection can be Sensitive to the Conversational Context [64.28043776806213]
  We construct and publicly release a dataset of 10,000 posts with two kinds of toxicity labels. We introduce a new task, context sensitivity estimation, which aims to identify posts whose perceived toxicity changes if the context is also considered.
  arXiv  Detail & Related papers  (2021-11-19T13:57:26Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• Toxicity Detection: Does Context Really Matter? [22.083682201142242]
  We find that context can amplify or mitigate the perceived toxicity of posts. Surprisingly, we also find no evidence that context actually improves the performance of toxicity classifiers. This points to the need for larger datasets of comments annotated in context.
  arXiv  Detail & Related papers  (2020-06-01T15:03:48Z)
• Deep Collaborative Embedding for information cascade prediction [58.90540495232209]
  We propose a novel model called Deep Collaborative Embedding (DCE) for information cascade prediction. We propose an auto-encoder based collaborative embedding framework to learn the node embeddings with cascade collaboration and node collaboration. The results of extensive experiments conducted on real-world datasets verify the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-01-18T13:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.