Auditing the Auditors: Does Community-based Moderation Get It Right?

• URL: http://arxiv.org/abs/2603.18053v1
• Date: Tue, 17 Mar 2026 21:58:13 GMT
• Title: Auditing the Auditors: Does Community-based Moderation Get It Right?
• Abstract summary: We analyze the consequences of consensus-based auditing in X's Community Notes.<n>Minority contributors' evaluations drift toward the majority, and their participation share falls on controversial topics.<n>We propose a two-stage auditing and aggregation algorithm that weights contributors by the stability of their past residuals rather than by agreement with the majority.
• Score: 0.5833117322405447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online social platforms increasingly rely on crowd-sourced systems to label misleading content at scale, but these systems must both aggregate users' evaluations and decide whose evaluations to trust. To address the latter, many platforms audit users by rewarding agreement with the final aggregate outcome, a design we term consensus-based auditing. We analyze the consequences of this design in X's Community Notes, which in September 2022 adopted consensus-based auditing that ties users' eligibility for participation to agreement with the eventual platform outcome. We find evidence of strategic conformity: minority contributors' evaluations drift toward the majority and their participation share falls on controversial topics, where independent signals matter most. We formalize this mechanism in a behavioral model in which contributors trade off private beliefs against anticipated penalties for disagreement. Motivated by these findings, we propose a two-stage auditing and aggregation algorithm that weights contributors by the stability of their past residuals rather than by agreement with the majority. The method first accounts for differences across content and contributors, and then measures how predictable each contributor's evaluations are relative to the latent-factor model. Contributors whose evaluations are consistently informative receive greater influence in aggregation, even when they disagree with the prevailing consensus. In the Community Notes data, this approach improves out-of-sample predictive performance while avoiding penalization of disagreement.

Related papers

• Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy [19.98446089883166]
  We contribute the largest dataset of its kind: four consultations spanning over 90k participants, 1M votes, and 22 languages.<n>Our experiments show that models with comparable predictive accuracy can differ substantially in the degree to which they preserve the collective structure.<n>This work aims to support the development of AI systems that scale deliberation without compromising the integrity of its democratic outcomes.
  arXiv  Detail & Related papers  (2026-09-02T15:23:00Z)
• Gaming Consensus: Coordinated Manipulation in Crowdsourced Fact-Checking [22.670031445884746]
  Crowdsourced fact-checking systems have been adopted by major social media companies such as X, Meta, TikTok and Google.<n>This work examines the core matrix factorization portion of these systems.<n>We find that up to 10.7% of lower quality notes could be manipulated above consensus thresholds using less than 10 ratings.
  arXiv  Detail & Related papers  (2026-07-02T07:46:44Z)
• Fairness Dynamics in Digital Economy Platforms with Biased Ratings [50.29721091981893]
  We study how digital platforms can perpetuate or counteract rating-based discrimination.<n>Our results demonstrate a fundamental trade-off between user experience and fairness.<n>Our results also provide evidence that intervening by tuning the demographics of the search results is a highly effective way of reducing unfairness.
  arXiv  Detail & Related papers  (2026-02-18T18:41:16Z)
• Distribution-Calibrated Inference time compute for Thinking LLM-as-a-Judge [5.855996386998925]
  Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level.<n>We study inference-time compute (ITC) for evaluators that generate n independent thinking-rating samples per item.
  arXiv  Detail & Related papers  (2025-12-02T18:46:47Z)
• Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems [29.596401271139797]
  We introduce Retentive Relevance, a novel content-level survey-based feedback measure.<n>Retentive Relevance directly assesses users' intent to return to the platform for similar content.<n>We show that Retentive Relevance significantly outperforms both engagement signals and other survey measures in predicting next-day retention.
  arXiv  Detail & Related papers  (2025-10-08T23:38:57Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Diverging Preferences: When do Annotators Disagree and do Models Know? [92.24651142187989]
  We develop a taxonomy of disagreement sources spanning 10 categories across four high-level classes. We find that the majority of disagreements are in opposition with standard reward modeling approaches. We develop methods for identifying diverging preferences to mitigate their influence on evaluation and training.
  arXiv  Detail & Related papers  (2024-10-18T17:32:22Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms [14.872177285434999]
  We ask queer communities for their positions on, and desires from, auditing processes. We organized a participatory workshop to critique and bias bounties from queer perspectives. We conclude by advocating for community ownership of bounties and complementing bounties with participatory processes.
  arXiv  Detail & Related papers  (2023-07-15T00:49:45Z)
• Debiasing Recommendation by Learning Identifiable Latent Confounders [49.16119112336605]
  Confounding bias arises due to the presence of unmeasured variables that can affect both a user's exposure and feedback. Existing methods either (1) make untenable assumptions about these unmeasured variables or (2) directly infer latent confounders from users' exposure. We propose a novel method, i.e., identifiable deconfounder (iDCF), which leverages a set of proxy variables to resolve the aforementioned non-identification issue.
  arXiv  Detail & Related papers  (2023-02-10T05:10:26Z)
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems [76.79703106646967]
  We develop a value-based assessment framework that visualizes closeness and tensions between values. We give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.
  arXiv  Detail & Related papers  (2022-05-09T19:28:32Z)
• A Robust Reputation-based Group Ranking System and its Resistance to Bribery [8.300507994596416]
  We propose a new reputation-based ranking system, utilizing multipartite ratingworks. We study its resistance to bribery and how to design optimal bribing strategies.
  arXiv  Detail & Related papers  (2020-04-13T22:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.