It is not enough to give your moderation rules to ChatGPT: Policy-as-Prompt Moderation and Its Potential Impacts on Community Governance

• URL: http://arxiv.org/abs/2607.12149v2
• Date: Wed, 15 Jul 2026 19:46:08 GMT
• Title: It is not enough to give your moderation rules to ChatGPT: Policy-as-Prompt Moderation and Its Potential Impacts on Community Governance
• Abstract summary: Social media companies are focusing more on community approaches, relying on volunteers to provide accurate information and make correct decisions.<n>In decentralized moderation, communities have always relied on volunteers, updated community guidelines, and internal discussions thereof.<n>For both content moderation paradigms, Artificial Intelligence (AI) seems like it could help ease moderation burdens of time, mental health, and accuracy.<n>One possible way to operationalize AI in content moderation is a policy-as-prompt'' approach, where the policy is formulated as a natural-language prompt and then passed to a large language model.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Content moderation practices and governance paradigms are changing rapidly, as fewer human moderators are deployed as `experts' by social media companies in a centralized manner. Instead, the companies are focusing more on community approaches, relying on volunteers to provide accurate information and make correct decisions. In decentralized moderation, communities have always relied on volunteers, updated community guidelines, and internal discussions thereof. For both content moderation paradigms, Artificial Intelligence (AI) seems like it could help ease moderation burdens of time, mental health, and accuracy. One possible way to operationalize AI in content moderation is a `policy-as-prompt'' approach, where the policy is formulated as a natural-language prompt and then passed to a large language model (LLM). This model then aids in moderation tasks. In this paper, we briefly lay out the technical and governance properties of this approach, and argue that its limitations lead to specific risks and harms that have to be addressed. Towards alleviating them, we lay out multiple considerations towards more effective prompt governance, but ultimately find that writing prompts alone is not appropriate for ensuring meaningful community governance.

Related papers

• SocraticPO: Policy Optimization via Interactive Guidance [64.7524628699057]
  Reinforcement learning (RL) for large language models usually supervises reasoning with scalar outcome rewards, such as binary correctness.<n>We propose textbfSocraticPO (Socratic Policy Optimization), a policy-optimization framework that augments RL rollouts with Socratic-style natural-language guidance.
  arXiv  Detail & Related papers  (2026-06-03T09:08:29Z)
• The Gray Area: Characterizing Moderator Disagreement on Reddit [4.508230455103701]
  One-in-seven moderation cases are disputed among moderators.<n>Almost half of all gray area cases involved automated moderation decisions.<n>We highlight the key role of expert human moderators in overseeing the moderation process.
  arXiv  Detail & Related papers  (2026-01-04T17:59:33Z)
• Community Moderation and the New Epistemology of Fact Checking on Social Media [124.26693978503339]
  Social media platforms have traditionally relied on independent fact-checking organizations to identify and flag misleading content.<n>X (formerly Twitter) and Meta have shifted towards community-driven content moderation by launching their own versions of crowd-sourced fact-checking.<n>We examine the current approaches to misinformation detection across major platforms, explore the emerging role of community-driven moderation, and critically evaluate both the promises and challenges of crowd-checking at scale.
  arXiv  Detail & Related papers  (2025-05-26T14:50:18Z)
• SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization [57.69385990442078]
  Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions)<n>Semantic entropy measures the diversity of meaning in multiple generated answers given a prompt and uses this to modulate the magnitude of policy updates.
  arXiv  Detail & Related papers  (2025-05-18T10:20:59Z)
• Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt [44.522343543870804]
  We are worried that such systems, which overlook enduring moral diversity, will spark resistance, erode trust, and destabilize our institutions.<n>This paper traces the underlying problem to an often-unstated Axiom of Rational Convergence: the idea that under ideal conditions, rational agents will converge in the limit of conversation on a single ethics.<n>Treating that premise as both optional and doubtful, we propose what we call the appropriateness framework: an alternative approach grounded in conflict theory, cultural evolution, multi-agent systems, and institutional economics.
  arXiv  Detail & Related papers  (2025-05-08T12:55:07Z)
• Media and responsible AI governance: a game-theoretic and LLM analysis [61.132523071109354]
  This paper investigates the interplay between AI developers, regulators, users, and the media in fostering trustworthy AI systems.<n>Using evolutionary game theory and large language models (LLMs), we model the strategic interactions among these actors under different regulatory regimes.
  arXiv  Detail & Related papers  (2025-03-12T21:39:38Z)
• Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis [48.14390493099495]
  How AI models should deal with political topics has been discussed, but it remains challenging and requires better governance.<n>This paper examines the governance of large language models through individual and collective deliberation, focusing on politically sensitive videos.
  arXiv  Detail & Related papers  (2024-09-15T03:17:38Z)
• Perceptions of Moderators as a Large-Scale Measure of Online Community Governance [13.80648276848838]
  We label 1.89 million posts and comments made on reddit over an 18 month period.<n>We identify types of communities where moderators are perceived particularly positively and negatively.<n>We show that strict rule enforcement is linked to more favorable perceptions of moderators of communities dedicated to certain topics, such as news communities, than others.
  arXiv  Detail & Related papers  (2024-01-29T22:43:12Z)
• "There Has To Be a Lot That We're Missing": Moderating AI-Generated Content on Reddit [5.202496456440801]
  We focus on online community moderators' experiences with AI-generated content (AIGC)<n>Our study finds that rules about AIGC are motivated by concerns about content quality, social dynamics, and governance challenges.<n>We find that, despite the absence of robust tools for detecting AIGC, moderators were able to somewhat limit the disruption it caused.
  arXiv  Detail & Related papers  (2023-11-21T16:15:21Z)
• Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation [75.86581380817464]
  A SimulST system generally includes two components: the pre-decision that aggregates the speech information and the policy that decides to read or write. This paper proposes to model the adaptive policy by adapting the Continuous Integrate-and-Fire (CIF) Compared with monotonic multihead attention (MMA), our method has the advantage of simpler computation, superior quality at low latency, and better generalization to long utterances.
  arXiv  Detail & Related papers  (2022-03-22T23:33:18Z)
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals [89.67840168694259]
  We propose to incorporate imagined subgoals into policy learning to facilitate learning of complex tasks. Imagined subgoals are predicted by a separate high-level policy, which is trained simultaneously with the policy and its critic. We evaluate our approach on complex robotic navigation and manipulation tasks and show that it outperforms existing methods by a large margin.
  arXiv  Detail & Related papers  (2021-07-01T15:30:59Z)
• Moderation Challenges in Voice-based Online Communities on Discord [24.417653462255448]
  Findings suggest that the affordances of voice-based online communities change what it means to moderate content and interactions. New ways to break rules that moderators of text-based communities find unfamiliar, such as disruptive noise and voice raiding. New moderation strategies are limited and often based on hearsay and first impressions, resulting in problems ranging from unsuccessful moderation to false accusations.
  arXiv  Detail & Related papers  (2021-01-13T18:43:22Z)
• PolicyKit: Building Governance in Online Communities [21.20591117254434]
  We present PolicyKit, a software infrastructure that empowers online community members to concisely author a wide range of governance procedures. We draw on political science theory to encode community governance into policies, or short imperative functions that specify a procedure for determining whether a user-initiated action can execute. We demonstrate the expressivity of PolicyKit through implementations of governance models such as a random jury deliberation, a multi-stage caucus, a reputation system, and a promotion procedure inspired by Wikipedia's Request for Adminship (RfA) process.
  arXiv  Detail & Related papers  (2020-08-10T16:17:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.