Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals

• URL: http://arxiv.org/abs/2603.03242v1
• Date: Tue, 03 Mar 2026 18:36:25 GMT
• Title: Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals
• Authors: Patrick Gerard, Svitlana Volkova,
• Abstract summary: We show that accepted responses occupy coherent, high-density regions that reflect community-specific norms.<n>We introduce density-guided response optimization (DGRO), a method that aligns language models to community norms without requiring explicit preference labels.
• Score: 0.8839687029212673
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models deployed in online communities must adapt to norms that vary across social, cultural, and domain-specific contexts. Prior alignment approaches rely on explicit preference supervision or predefined principles, which are effective for well-resourced settings but exclude most online communities -- particularly those without institutional backing, annotation infrastructure, or organized around sensitive topics -- where preference elicitation is costly, ethically fraught, or culturally misaligned. We observe that communities already express preferences implicitly through what content they accept, engage with, and allow to persist. We show that this acceptance behavior induces measurable geometric structure in representation space: accepted responses occupy coherent, high-density regions that reflect community-specific norms, while rejected content falls in sparser or misaligned areas. We operationalize this structure as an implicit preference signal for alignment and introduce density-guided response optimization (DGRO), a method that aligns language models to community norms without requiring explicit preference labels. Using labeled preference data, we demonstrate that local density recovers pairwise community judgments, indicating that geometric structure encodes meaningful preference signal. We then apply DGRO in annotation-scarce settings across diverse communities spanning platform, topic, and language. DGRO-aligned models consistently produce responses preferred by human annotators, domain experts, and model-based judges over supervised and prompt-based baselines. We position DGRO as a practical alignment alternative for communities where explicit preference supervision is unavailable or misaligned with situated practices, and discuss the implications and risks of learning from emergent acceptance behavior.

Related papers

• Position: General Alignment Has Hit a Ceiling; Edge Alignment Must Be Taken Seriously [51.03213216886717]
  We take the position that the dominant paradigm of General Alignment reaches a structural ceiling in settings with conflicting values.<n>We introduce Edge Alignment as a distinct approach in which systems preserve multi dimensional value structure.
  arXiv  Detail & Related papers  (2026-02-23T16:51:43Z)
• Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment [6.428964221372943]
  We propose Geometric Anchor Preference Optimization (GAPO), which replaces the fixed reference with a dynamic, geometry-aware anchor.<n>GAPO consistently improves robustness while matching or improving performance on standard LLM alignment and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-02-04T00:40:21Z)
• Where Norms and References Collide: Evaluating LLMs on Normative Reasoning [3.8431932182760296]
  Embodied agents, such as robots, will need to interact in situated environments where successful communication often depends on reasoning over social norms.<n>It remains unclear whether Large Language Models (LLMs) can support this kind of reasoning.<n>We introduce SNIC (Situated Norms in Context), a human-validated diagnostic testbed designed to probe how well state-of-the-art LLMs can extract and utilize normative principles relevant to NBRR.
  arXiv  Detail & Related papers  (2026-02-03T01:23:22Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  We argue that replacing parameters with preferences represents a more scalable and privacy-preserving future.<n>We propose MoR, a federated alignment framework based on GRPO with Mixture-of-Rewards for heterogeneous VLMs.<n>MoR consistently outperforms federated alignment baselines in generalization, robustness, and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-01-31T03:11:51Z)
• Latent Adversarial Regularization for Offline Preference Optimization [21.271580780278473]
  We introduce GANPO, which achieves latent-space regularization by penalizing divergence between internal representations of a policy model and a reference model.<n>Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization.
  arXiv  Detail & Related papers  (2026-01-29T18:21:57Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces [3.203159763233367]
  We introduce the Local Intersectional Visual Spaces dataset, a benchmark for multi-criteria alignment.<n>The dataset encodes 37,710 pairwise comparisons across 13,462 images, structured along six criteria.<n>We fine-tune Stable Diffusion XL to reflect multi-criteria spatial preferences and evaluate the LIVS dataset and the fine-tuned model.
  arXiv  Detail & Related papers  (2025-02-27T19:18:37Z)
• ComPO: Community Preferences for Language Model Personalization [122.54846260663922]
  ComPO is a method to personalize preference optimization in language models. We collect and release ComPRed, a question answering dataset with community-level preferences from Reddit.
  arXiv  Detail & Related papers  (2024-10-21T14:02:40Z)
• An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Context [109.49663559151377]
  Existing works on adaptation to unknown environment contexts either assume the contexts are the same for the whole episode or assume the context variables are Markovian. In this paper, we propose a textittextbfSegmented textbfContext textbfBelief textbfAugmented textbfDeep(SeCBAD) RL method. Our method can jointly infer the belief distribution over latent context with the posterior over segment length and perform more accurate belief context inference with observed data within the current
  arXiv  Detail & Related papers  (2022-12-24T13:43:39Z)
• Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding [214.8003571700285]
  Weakly supervised Referring Expression Grounding (REG) aims to ground a particular target in an image described by a language expression. We design an entity-enhanced adaptive reconstruction network (EARN) EARN includes three modules: entity enhancement, adaptive grounding, and collaborative reconstruction.
  arXiv  Detail & Related papers  (2022-07-18T05:30:45Z)
• Detecting Community Sensitive Norm Violations in Online Conversations [21.892867827127603]
  We focus on a more complete spectrum of community norms and their violations in the local conversational and global community contexts. We introduce a series of models that use this data to develop context- and community-sensitive norm violation detection.
  arXiv  Detail & Related papers  (2021-10-09T00:39:35Z)
• Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation [78.38321096371106]
  Prior arts assume concurrent access to both labeled source and unlabeled target, making them unsuitable for scenarios demanding source-free adaptation. In this work, we enable source-free DA by partitioning the task into two: a) source-only domain generalization and b) source-free target adaptation. We introduce a novel conditional prior-enforcing auto-encoder that discourages spatial irregularities, thereby enhancing the pseudo-label quality.
  arXiv  Detail & Related papers  (2021-08-25T14:18:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.