Coherence Maximization Improves Pluralistic Alignment

• URL: http://arxiv.org/abs/2606.03110v2
• Date: Wed, 03 Jun 2026 08:07:14 GMT
• Title: Coherence Maximization Improves Pluralistic Alignment
• Abstract summary: Internal Coherence Maximization (ICM) infers labels by maximizing their mutual predictability.<n>More coherent examples generalize substantially better than incoherent ones.<n>These results identify coherence as a key design principle for scalable value specification.
• Score: 6.466328345684073
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Aligning AI systems with diverse human values requires value specifications grounded in concrete examples, but generating such examples without extensive human supervision remains an open challenge. We investigate what makes these examples effective, using Internal Coherence Maximization (ICM) -- which infers labels by maximizing their mutual predictability -- to generate persona-specific examples that steer a model toward a target group's values, without human supervision. Across four benchmarks spanning classification, preference, and open-ended generation, ICM-inferred in-context examples match the performance of gold labels. Crucially, coherence matters beyond individual label accuracy: with accuracy held constant, more coherent examples generalize substantially better than incoherent ones. For personas underrepresented in pretraining data, targeted human feedback on the questions where the model is least certain about a persona's values yields better generalization than the same number of labels on arbitrary questions. These results identify coherence as a key design principle for scalable value specification, leveraging the diverse human perspectives already encoded in pretrained language models.

Related papers

• GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation [74.44235943118994]
  GenRubric is a self-evolving framework that improves rubric generation from unlabeled queries.<n>We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality.<n> Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.
  arXiv  Detail & Related papers  (2026-08-30T15:39:39Z)
• Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Models [43.01088871836861]
  COUPLE is a COUnterfactual reasoning framework for PLuralistic valuE alignment.<n>It features complex interdependency and prioritization among features, as well as the causal relationship between high-level value dimensions and behaviors.<n>Benefitting from explicit causal modeling, COUPLE also provides better interpretability.
  arXiv  Detail & Related papers  (2025-10-21T11:12:45Z)
• MoVa: Towards Generalizable Classification of Human Morals and Values [57.93595662296688]
  MoVa is a well-documented suite of resources for generalizable classification of human morals and values.<n>The data and methods in MoVa can facilitate many fine-grained interpretations of human and machine communication.
  arXiv  Detail & Related papers  (2025-09-29T02:56:27Z)
• EigenBench: A Comparative Behavioral Measure of Value Alignment [0.28707625120094377]
  EigenBench is a black-box method for benchmarking language models' values.<n>It is designed to quantify subjective traits for which reasonable judges may disagree on the correct label.<n>It can recover model rankings on the GPQA benchmark without access to objective labels.
  arXiv  Detail & Related papers  (2025-09-02T04:14:26Z)
• Preference Learning for AI Alignment: a Causal Perspective [55.2480439325792]
  We frame this problem in a causal paradigm, providing the rich toolbox of causality to identify persistent challenges.<n>Inheriting from the literature of causal inference, we identify key assumptions necessary for reliable generalisation.<n>We illustrate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness.
  arXiv  Detail & Related papers  (2025-06-06T10:45:42Z)
• HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation [44.973773675725674]
  Large multimodal models (LMMs) have achieved impressive performance on vision tasks such as visual question answering (VQA), image captioning, and visual grounding.<n>HumaniBench is a benchmark comprising 32,000 real-world image-question pairs and an accompanying evaluation suite.<n>It assesses LMMs across seven key alignment principles: fairness, ethics, empathy, inclusivity, reasoning, robustness, and multilinguality.
  arXiv  Detail & Related papers  (2025-05-16T17:09:44Z)
• Self-supervised Attribute-aware Dynamic Preference Ranking Alignment [21.572914948747748]
  We propose textbfSelf-supervised textbfAttribute-aware textbfdynamic textbfpreference textbfranking, called shortname.<n>It quantifies preference differences between responses based on Attribute-Perceptual Distance Factors (APDF) and dynamically determines the list-wise alignment order.<n>It achieves fine-grained preference difference learning and enables precise alignment with the optimal one.
  arXiv  Detail & Related papers  (2025-02-15T08:20:42Z)
• Probably Approximately Precision and Recall Learning [60.00180898830079]
  A key challenge in machine learning is the prevalence of one-sided feedback.<n>We introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels.<n>We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• MAP: Multi-Human-Value Alignment Palette [22.74688073469946]
  We develop a novel, first-principle approach called Multi-Human-Value Alignment Palette (MAP) MAP navigates the alignment across multiple human values in a structured and reliable way. We conduct a detailed theoretical analysis of MAP by quantifying the trade-offs between values, the sensitivity to constraints, the fundamental connection between multi-value alignment and sequential alignment, and proving that linear weighted rewards are sufficient for multi-value alignment.
  arXiv  Detail & Related papers  (2024-10-24T23:16:39Z)
• Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypes [6.30817290125825]
  We introduce the GUS-Net Framework, comprising the GUS dataset and a multi-label token-level detector for span-level analysis of social bias.<n>The GUS dataset contains 3,739 unique snippets across multiple domains, with over 69,000 token-level annotations.<n>We formulate bias detection as multi-label token-level classification and benchmark both encoder-based models and decoder-based large language models.
  arXiv  Detail & Related papers  (2024-10-10T21:51:22Z)
• Self-Training with Pseudo-Label Scorer for Aspect Sentiment Quad Prediction [54.23208041792073]
  Aspect Sentiment Quad Prediction (ASQP) aims to predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review. A key challenge in the ASQP task is the scarcity of labeled data, which limits the performance of existing methods. We propose a self-training framework with a pseudo-label scorer, wherein a scorer assesses the match between reviews and their pseudo-labels.
  arXiv  Detail & Related papers  (2024-06-26T05:30:21Z)
• Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives! [4.478789600295492]
  We consider the interplay of individual and shared perspectives in natural language processing. We investigate the extent to which architectures that include layers to model the relations between different annotators are beneficial for predicting single-annotator labels. Our findings indicate that approaches to subjectivity can benefit from relating individual perspectives.
  arXiv  Detail & Related papers  (2023-11-06T14:47:48Z)
• Probabilistic Test-Time Generalization by Variational Neighbor-Labeling [62.158807685159736]
  This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. Probability pseudo-labeling of target samples to generalize the source-trained model to the target domain at test time. Variational neighbor labels that incorporate the information of neighboring target samples to generate more robust pseudo labels.
  arXiv  Detail & Related papers  (2023-07-08T18:58:08Z)
• Heterogeneous Value Alignment Evaluation for Large Language Models [91.96728871418]
  Large Language Models (LLMs) have made it crucial to align their values with those of humans. We propose a Heterogeneous Value Alignment Evaluation (HVAE) system to assess the success of aligning LLMs with heterogeneous values.
  arXiv  Detail & Related papers  (2023-05-26T02:34:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.