PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media

• URL: http://arxiv.org/abs/2605.17187v1
• Date: Sat, 16 May 2026 22:52:11 GMT
• Title: PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media
• Abstract summary: Social media are shifting towards pluralism -- community-governed platforms where groups define their own norms.<n>We formalize the task as a multiple-choice problem, mirroring how human moderators operate in the real world.<n>We introduce PluRule, a multimodal, multilingual benchmark for detecting 13,371 rule violations across 1,989 Reddit communities spanning 2,885 rules in 9 languages.
• Score: 5.451879555600225
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Social media are shifting towards pluralism -- community-governed platforms where groups define their own norms. What violates rules in one community may be perfectly acceptable in another. Can AI models help moderate such pluralistic communities? We formalize the task as a multiple-choice problem, mirroring how human moderators operate in the real world: given a comment and its surrounding context, identify which specific rule, if any, is violated. We introduce PluRule, a multimodal, multilingual benchmark for detecting 13,371 rule violations across 1,989 Reddit communities spanning 2,885 rules in 9 languages. Using this benchmark, we show that state-of-the-art vision-language models struggle significantly: even GPT-5.2 with high reasoning performs only slightly better than a trivial baseline. We also find that bigger models and increased context provide marginal gains, and universal rules like civility and self-promotion are easier to detect. Our results show that moderation of pluralistic communities on social media is a fundamental challenge for language models. Our code and benchmark are publicly available.

Related papers

• Do Agents Repair When Challenged -- or Just Reply? Challenge, Repair, and Public Correction in a Deployed Agent Forum [9.934280404394636]
  We compare Moltbook, a large language model (LLM) agent forum, with five matched Reddit communities.<n>Relative to Reddit, Moltbook discussions are roughly ten times less threaded, leaving far fewer chances for challenge and response.<n>A non-challenge baseline within Reddit suggests this gap is linked to challenge, not simply deeper threading.
  arXiv  Detail & Related papers  (2026-04-01T06:03:28Z)
• Asking For It: Question-Answering for Predicting Rule Infractions in Online Content Moderation [1.803599876087764]
  ModQ is a novel question-answering framework for rule-sensitive content moderation.<n>We implement two model variants and train them on large-scale datasets from Reddit and Lemmy.<n>Both models outperform state-of-the-art baselines in identifying moderation-relevant rule violations.
  arXiv  Detail & Related papers  (2025-10-07T18:11:27Z)
• Reddit Rules and Rulers: Quantifying the Link Between Rules and Perceptions of Governance across Thousands of Communities [13.80648276848838]
  We conduct the largest-to-date analysis of rules on Reddit, collecting a set of 67,545 unique rules across 5,225 communities.<n>More than just a point-in-time study, our work measures how communities change their rules over a 5+ year period.<n>We are the first to identify the rules most strongly associated with positive community perceptions of governance.
  arXiv  Detail & Related papers  (2025-01-24T01:26:41Z)
• A Roadmap to Pluralistic Alignment [49.29107308098236]
  We propose a roadmap to pluralistic alignment, specifically using language models as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems. We argue that current alignment techniques may be fundamentally limited for pluralistic AI.
  arXiv  Detail & Related papers  (2024-02-07T18:21:17Z)
• Perceptions of Moderators as a Large-Scale Measure of Online Community Governance [13.80648276848838]
  We label 1.89 million posts and comments made on reddit over an 18 month period.<n>We identify types of communities where moderators are perceived particularly positively and negatively.<n>We show that strict rule enforcement is linked to more favorable perceptions of moderators of communities dedicated to certain topics, such as news communities, than others.
  arXiv  Detail & Related papers  (2024-01-29T22:43:12Z)
• SADAS: A Dialogue Assistant System Towards Remediating Norm Violations in Bilingual Socio-Cultural Conversations [56.31816995795216]
  Socially-Aware Dialogue Assistant System (SADAS) is designed to ensure that conversations unfold with respect and understanding. Our system's novel architecture includes: (1) identifying the categories of norms present in the dialogue, (2) detecting potential norm violations, (3) evaluating the severity of these violations, and (4) implementing targeted remedies to rectify the breaches.
  arXiv  Detail & Related papers  (2024-01-29T08:54:21Z)
• Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms [65.17491295329991]
  We construct a new benchmark for studying visual-grounded commonsense norms: NORMLENS. We find that state-of-the-art model judgments and explanations are not well-aligned with human annotation. We present a new approach to better align models with humans by distilling social commonsense knowledge from large language models.
  arXiv  Detail & Related papers  (2023-10-16T14:00:07Z)
• Analyzing Norm Violations in Live-Stream Chat [49.120561596550395]
  We study the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. Our results show that appropriate contextual information can boost moderation performance by 35%.
  arXiv  Detail & Related papers  (2023-05-18T05:58:27Z)
• Language Models are Multilingual Chain-of-Thought Reasoners [83.37148309771378]
  We introduce the Multilingual Grade School Math benchmark, by manually translating 250 grade-school math problems into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale. We show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment.
  arXiv  Detail & Related papers  (2022-10-06T17:03:34Z)
• Norm Participation Grounds Language [16.726800816202033]
  I propose a different, and more wide-ranging conception of how grounding should be understood: What grounds language is its normative nature. There are standards for doing things right, these standards are public and authoritative, while at the same time acceptance of authority can be disputed and negotiated. What grounds language, then, is the determined use that language users make of it, and what it is grounded in is the community of language users.
  arXiv  Detail & Related papers  (2022-06-06T20:21:59Z)
• Learning Symbolic Rules for Reasoning in Quasi-Natural Language [74.96601852906328]
  We build a rule-based system that can reason with natural language input but without the manual construction of rules. We propose MetaQNL, a "Quasi-Natural" language that can express both formal logic and natural language sentences. Our approach achieves state-of-the-art accuracy on multiple reasoning benchmarks.
  arXiv  Detail & Related papers  (2021-11-23T17:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.