CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks

• URL: http://arxiv.org/abs/2601.13669v1
• Date: Tue, 20 Jan 2026 07:10:50 GMT
• Title: CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks
• Abstract summary: We introduce CommunityBench, the first large-scale benchmark for community-level alignment evaluation.<n>We conduct a comprehensive evaluation of various foundation models on CommunityBench, revealing that current LLMs exhibit limited capacity to model community-specific preferences.
• Score: 32.59328811602595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) alignment ensures model behaviors reflect human value. Existing alignment strategies primarily follow two paths: one assumes a universal value set for a unified goal (i.e., one-size-fits-all), while the other treats every individual as unique to customize models (i.e., individual-level). However, assuming a monolithic value space marginalizes minority norms, while tailoring individual models is prohibitively expensive. Recognizing that human society is organized into social clusters with high intra-group value alignment, we propose community-level alignment as a "middle ground". Practically, we introduce CommunityBench, the first large-scale benchmark for community-level alignment evaluation, featuring four tasks grounded in Common Identity and Common Bond theory. With CommunityBench, we conduct a comprehensive evaluation of various foundation models on CommunityBench, revealing that current LLMs exhibit limited capacity to model community-specific preferences. Furthermore, we investigate the potential of community-level alignment in facilitating individual modeling, providing a promising direction for scalable and pluralistic alignment.

Related papers

• GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs [22.02836146038364]
  We present GroupToM-Bench, the first multimodal benchmark for group-level Theory of Mind (ToM) reasoning.<n>We build around a causal chain spanning micro-level BDI states (belief, desire, intention), meso-level group tension and structural constraints, and macro-level outcome prediction and attribution.<n>Experiments reveal a gap between current models and human baselines, highlighting a failure to process social structures and non-linear collective dynamics.
  arXiv  Detail & Related papers  (2026-06-02T20:06:32Z)
• Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment [75.88231994139132]
  We show that even state-of-the-art models like GPT-4.1 achieve only 57.4% accuracy in predicting modal preferences.<n>We find significant pre-existing performance biases, where models better emulate young, male, Chinese, and Christian personas.
  arXiv  Detail & Related papers  (2026-04-14T15:06:13Z)
• LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations [63.478832978278014]
  Large language models (LLMs) are increasingly deployed as autonomous agents, yet evaluations focus primarily on task success rather than cultural appropriateness or evaluator reliability.<n>We introduce LiveCultureBench, a multi-cultural, dynamic benchmark that embeds LLMs as agents in a simulated town and evaluates them on both task completion and adherence to socio-cultural norms.
  arXiv  Detail & Related papers  (2026-03-02T15:04:16Z)
• A Theoretical Framework for Adaptive Utility-Weighted Benchmarking [0.0]
  This paper introduces a theoretical framework that reconceptualizes benchmarking as a multilayer, adaptive network linking evaluation metrics, model components, and stakeholder groups through weighted interactions.<n>Using conjoint-derived utilities and a human-in-the-loop update rule, we formalize how human tradeoffs can be embedded into benchmark structure and how benchmarks can evolve dynamically while preserving stability and interpretability.
  arXiv  Detail & Related papers  (2026-02-12T19:33:47Z)
• Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning [4.735670734773144]
  Value-aware AI should recognise human values and adapt to the value systems (value-based preferences) of different users.<n>We propose algorithms for learning models of value alignment and value systems for a society of agents.
  arXiv  Detail & Related papers  (2026-02-09T16:06:36Z)
• A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs [2.840505903487544]
  This paper addresses the challenge of aligning large language models (LLMs) with diverse human preferences within federated learning environments.<n>We introduce a comprehensive evaluation framework that assesses the trade-off between alignment quality and fairness when using different aggregation strategies for human preferences.
  arXiv  Detail & Related papers  (2025-12-09T16:39:32Z)
• MF-LLM: Simulating Population Decision Dynamics via a Mean-Field Large Language Model Framework [53.82097200295448]
  Mean-Field LLM (MF-LLM) is first to incorporate mean field theory into social simulation.<n>MF-LLM models bidirectional interactions between individuals and the population through an iterative process.<n> IB-Tune is a novel fine-tuning method inspired by the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-04-30T12:41:51Z)
• LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces [3.203159763233367]
  We introduce the Local Intersectional Visual Spaces dataset, a benchmark for multi-criteria alignment.<n>The dataset encodes 37,710 pairwise comparisons across 13,462 images, structured along six criteria.<n>We fine-tune Stable Diffusion XL to reflect multi-criteria spatial preferences and evaluate the LIVS dataset and the fine-tuned model.
  arXiv  Detail & Related papers  (2025-02-27T19:18:37Z)
• CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution [74.41064280094064]
  textbfJudger-1 is the first open-source textbfall-in-one judge LLM. CompassJudger-1 is a general-purpose LLM that demonstrates remarkable versatility. textbfJudgerBench is a new benchmark that encompasses various subjective evaluation tasks.
  arXiv  Detail & Related papers  (2024-10-21T17:56:51Z)
• COMMUNITY-CROSS-INSTRUCT: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communities [5.0261645603931475]
  We introduce Community-Cross-Instruct, an unsupervised framework for aligning large language models to online communities to elicit beliefs. We demonstrate the method's utility in accurately representing political and diet communities on Reddit.
  arXiv  Detail & Related papers  (2024-06-17T20:20:47Z)
• Universal Semi-supervised Model Adaptation via Collaborative Consistency Training [92.52892510093037]
  We introduce a realistic and challenging domain adaptation problem called Universal Semi-supervised Model Adaptation (USMA) We propose a collaborative consistency training framework that regularizes the prediction consistency between two models. Experimental results demonstrate the effectiveness of our method on several benchmark datasets.
  arXiv  Detail & Related papers  (2023-07-07T08:19:40Z)
• Heterogeneous Value Alignment Evaluation for Large Language Models [91.96728871418]
  Large Language Models (LLMs) have made it crucial to align their values with those of humans. We propose a Heterogeneous Value Alignment Evaluation (HVAE) system to assess the success of aligning LLMs with heterogeneous values.
  arXiv  Detail & Related papers  (2023-05-26T02:34:20Z)
• This Must Be the Place: Predicting Engagement of Online Communities in a Large-scale Distributed Campaign [70.69387048368849]
  We study the behavior of communities with millions of active members. We develop a hybrid model, combining textual cues, community meta-data, and structural properties. We demonstrate the applicability of our model through Reddit's r/place a large-scale online experiment.
  arXiv  Detail & Related papers  (2022-01-14T08:23:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.