Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy

• URL: http://arxiv.org/abs/2609.02990v2
• Date: Thu, 10 Sep 2026 11:28:12 GMT
• Title: Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy
• Abstract summary: We contribute the largest dataset of its kind: four consultations spanning over 90k participants, 1M votes, and 22 languages.<n>Our experiments show that models with comparable predictive accuracy can differ substantially in the degree to which they preserve the collective structure.<n>This work aims to support the development of AI systems that scale deliberation without compromising the integrity of its democratic outcomes.
• Score: 19.98446089883166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To scale up collective decision-making, participatory democracy platforms such as Polis and Remesh enable online deliberation among thousands of participants. However, at this scale, participants cannot review every opinion submitted by others, producing highly sparse voting data that misrepresent patterns of consensus, conflict, and minority support. Platforms therefore increasingly rely on Preference Inference (PI) models to predict missing votes. Yet this automation is not neutral: inferred preferences can artificially amplify, suppress, or reorder existing patterns of support, ultimately reshaping how the outcomes of a deliberation are interpreted. More generally, we lack a systematic understanding of how existing PI methods affect the collective preference landscape. To address this gap, we benchmark several existing PI approaches in this context. Moving beyond conventional user-centric evaluations centered on the accuracy of individual predictions, we introduce a collective-centric evaluation framework that measures whether inferred votes preserve salient properties of the broader preference landscape. We further contribute the largest multilingual dataset of its kind: four consultations spanning over 90k participants, 1M votes, and 22 languages. Our experiments show that models with comparable predictive accuracy can differ substantially in the degree to which they preserve the collective structure. These results demonstrate that accuracy alone is insufficient for evaluating PI in democratic settings. By contributing a novel comprehensive and collective-centric evaluation benchmark for the task of PI, this work aims to support the development of AI systems that scale deliberation without compromising the integrity of its democratic outcomes.

Related papers

• To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias [61.40435736418359]
  We introduce a unified framework that standardizes heterogeneous benchmarks to contrast isolated demographic assessments with forced-choice comparative settings.<n>We show that comparative settings act as aggressive catalysts for latent discrimination, a shift primarily driven by underspecified contexts.<n>Finally, we demonstrate that this comparative prejudice is a generalized phenomenon that scales positively with model size.
  arXiv  Detail & Related papers  (2026-06-23T13:53:50Z)
• Auditing the Auditors: Does Community-based Moderation Get It Right? [0.5833117322405447]
  We analyze the consequences of consensus-based auditing in X's Community Notes.<n>Minority contributors' evaluations drift toward the majority, and their participation share falls on controversial topics.<n>We propose a two-stage auditing and aggregation algorithm that weights contributors by the stability of their past residuals rather than by agreement with the majority.
  arXiv  Detail & Related papers  (2026-03-17T21:58:13Z)
• Algorithmic Approaches to Opinion Selection for Online Deliberation: A Comparative Study [1.5267938856942276]
  In online deliberation platforms, algorithmic selection is increasingly used to automate the selection process.<n>It remains unclear how each approach influences desired democratic criteria such as proportional representation.<n>We propose a novel algorithm that incorporates both diversity and a balanced notion of representation in the selection strategy.
  arXiv  Detail & Related papers  (2026-02-17T09:03:26Z)
• Democratic Preference Alignment via Sortition-Weighted RLHF [0.0]
  We introduce DemPO, a framework that applies algorithmic sortition, the same mechanism used to construct citizen assemblies, to preference based fine tuning.<n>We evaluate Llama models from one billion to eight billion parameters fine tuned under each scheme.<n>These results demonstrate that enforcing demographic representativeness at the preference collection stage, rather than post hoc correction, yields models whose behavior better reflects values elicited from representative publics.
  arXiv  Detail & Related papers  (2026-02-04T22:59:24Z)
• Beyond Marginal Distributions: A Framework to Evaluate the Representativeness of Demographic-Aligned LLMs [13.630995219491972]
  We propose a framework for evaluating the representativeness of aligned models.<n>We show the value of our evaluation scheme by comparing two model steering techniques.<n>We conclude that representativeness is a distinct aspect of value alignment.
  arXiv  Detail & Related papers  (2026-01-22T08:45:55Z)
• Snowveil: A Framework for Decentralised Preference Discovery [0.0]
  This paper introduces Decentralised Preference Discovery (DPD), the problem of determining the collective will of an electorate under constraints of censorship resistance, partial information, and asynchronous communication.<n>We propose Snowveil, a novel framework for this task. Snowveil uses an iterative, gossip-based protocol where voters repeatedly sample the preferences of a small, random subset of the electorate to progressively converge on a collective outcome.<n>We develop a multi-level analytical method to show that the system almost surely converges to a stable, single-winner in finite time, a process that can then be iterated to construct a set of winning
  arXiv  Detail & Related papers  (2025-12-20T17:31:55Z)
• hyperFA*IR: A hypergeometric approach to fair rankings with finite candidate pool [0.0]
  We present hyperFA*IR, a framework for assessing and enforcing fairness in rankings drawn from a finite set of candidates.<n>It relies on a generative process based on the hypergeometric distribution, which models real-world scenarios by sampling without replacement from fixed group sizes.<n>We also propose a Monte Carlo-based algorithm that efficiently detects unfair rankings by avoiding computationally expensive parameter tuning.
  arXiv  Detail & Related papers  (2025-06-17T09:45:08Z)
• Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework [7.065259679465175]
  We develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences.<n>We propose a soft-max relaxation method that smoothly trade-offs population-proportional alignment with the selection of the Condorcet winner.
  arXiv  Detail & Related papers  (2025-06-05T22:15:07Z)
• Re-evaluating Open-ended Evaluation of Large Language Models [50.23008729038318]
  We show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental.<n>We propose evaluation as a 3-player game, and introduce novel game-theoretic solution concepts to ensure robustness to redundancy.
  arXiv  Detail & Related papers  (2025-02-27T15:07:47Z)
• The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models [91.86718720024825]
  We center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias.<n>Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning.<n>We conclude with recommendations tailored to DPO and broader alignment practices.
  arXiv  Detail & Related papers  (2024-11-06T06:50:50Z)
• Diverging Preferences: When do Annotators Disagree and do Models Know? [92.24651142187989]
  We develop a taxonomy of disagreement sources spanning 10 categories across four high-level classes. We find that the majority of disagreements are in opposition with standard reward modeling approaches. We develop methods for identifying diverging preferences to mitigate their influence on evaluation and training.
  arXiv  Detail & Related papers  (2024-10-18T17:32:22Z)
• Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis [48.14390493099495]
  How AI models should deal with political topics has been discussed, but it remains challenging and requires better governance.<n>This paper examines the governance of large language models through individual and collective deliberation, focusing on politically sensitive videos.
  arXiv  Detail & Related papers  (2024-09-15T03:17:38Z)
• Aligning Large Language Models by On-Policy Self-Judgment [49.31895979525054]
  Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model (RM) for on-policy learning. We present a novel alignment framework, SELF-JUDGE, that does on-policy learning and is parameter efficient. We show that the rejecting sampling by itself can improve performance further without an additional evaluator.
  arXiv  Detail & Related papers  (2024-02-17T11:25:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.