VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

• URL: http://arxiv.org/abs/2603.18113v1
• Date: Wed, 18 Mar 2026 14:05:51 GMT
• Title: VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models
• Abstract summary: We propose VC-soup, a data filtering and parameter merging framework grounded in value-consistent learning.<n>We show that VC-soup effectively mitigates conflicts and consistently outperforms existing multi-value alignment methods.
• Score: 26.480803729157945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models (LLMs) increasingly shape content generation, interaction, and decision-making across the Web, aligning them with human values has become a central objective in trustworthy AI. This challenge becomes even more pronounced when aligning multiple, potentially conflicting human values. Although recent approaches, such as reward reweighting, prompt-based supervised fine-tuning, and model merging, attempt to tackle multi-value alignment, they still face two major limitations: (1) training separate models for each value combination is prohibitively expensive; (2) value conflicts substantially degrade alignment performance. These limitations make it difficult to achieve favorable trade-offs across diverse human values. To address these challenges, we revisit multi-value alignment from the perspective of value consistency in data and propose VC-soup, a data filtering and parameter merging framework grounded in value-consistent learning. We first design a value consistency metric based on the cosine similarity between the reward-gap vector of each preference pair and an all-ones vector, which quantifies its cross-value coherence. We then filter out low-consistency preference pairs in each value dataset and train on the remaining data to obtain smooth, value-consistent policy models that better preserve linear mode connectivity. Finally, we linearly combine these policies and apply Pareto filtering across values to obtain solutions with balanced multi-value performance. Extensive experiments and theoretical analysis demonstrate that VC-soup effectively mitigates conflicts and consistently outperforms existing multi-value alignment methods.

Related papers

• Contextual Value Alignment via Multilayer Combinatorial Fusion [5.97790552623697]
  We propose a framework that utilizes multilayer fusion for contextual value alignment (MCF-CVA)<n>At the first layer of the framework, it instantiates multiple moral agents, each fine-tuned to represent a distinctive value.<n>The MCF-CVA framework leverages cognitive diversity between agents to mitigate conflicts and redundancies across multiple agents, producing responses that better reflect contextual human values.
  arXiv  Detail & Related papers  (2026-08-07T16:22:33Z)
• ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation [34.173549610331385]
  Model merging aims to combine multiple task-specific expert models into a single model.<n>Interference among experts, especially when they are trained on different objectives, often leads to significant performance degradation.<n>acem is an Adaptive Covariance Estimation framework that effectively mitigates inter-task interference.
  arXiv  Detail & Related papers  (2026-03-03T12:53:04Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation [23.41040153806061]
  We propose a novel framework called Multi-Value Alignment (MVA)<n>It mitigates alignment caused by parameter interference among diverse human values by minimizing their mutual information.<n>MVA consistently outperforms existing baselines in aligning LLMs with multiple human values.
  arXiv  Detail & Related papers  (2025-11-15T13:33:26Z)
• Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Models [43.01088871836861]
  COUPLE is a COUnterfactual reasoning framework for PLuralistic valuE alignment.<n>It features complex interdependency and prioritization among features, as well as the causal relationship between high-level value dimensions and behaviors.<n>Benefitting from explicit causal modeling, COUPLE also provides better interpretability.
  arXiv  Detail & Related papers  (2025-10-21T11:12:45Z)
• Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation [49.98025799046136]
  We introduce Merge-And-GuidE, a two-stage framework that leverages model merging for guided decoding.<n>In Stage 1, MAGE resolves a compatibility problem between the guidance and base models.<n>In Stage 2, we merge explicit and implicit value models into a unified guidance proxy, which then steers the decoding of the base model from Stage 1.
  arXiv  Detail & Related papers  (2025-10-04T11:10:07Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• Principled Multimodal Representation Learning [99.53621521696051]
  Multimodal representation learning seeks to create a unified representation space by integrating diverse data modalities.<n>Recent advances have investigated the simultaneous alignment of multiple modalities, yet several challenges remain.<n>We propose Principled Multimodal Representation Learning (PMRL), a novel framework that achieves simultaneous alignment of multiple modalities.
  arXiv  Detail & Related papers  (2025-07-23T09:12:25Z)
• From Abstract to Actionable: Pairwise Shapley Values for Explainable AI [0.8192907805418583]
  We propose Pairwise Shapley Values, a novel framework that grounds feature attributions in explicit, human-relatable comparisons.<n>Our method introduces pairwise reference selection combined with single-value imputation to deliver intuitive, model-agnostic explanations.<n>We demonstrate that Pairwise Shapley Values enhance interpretability across diverse regression and classification scenarios.
  arXiv  Detail & Related papers  (2025-02-18T04:20:18Z)
• MAP: Multi-Human-Value Alignment Palette [22.74688073469946]
  We develop a novel, first-principle approach called Multi-Human-Value Alignment Palette (MAP) MAP navigates the alignment across multiple human values in a structured and reliable way. We conduct a detailed theoretical analysis of MAP by quantifying the trade-offs between values, the sensitivity to constraints, the fundamental connection between multi-value alignment and sequential alignment, and proving that linear weighted rewards are sufficient for multi-value alignment.
  arXiv  Detail & Related papers  (2024-10-24T23:16:39Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning [64.05646120624287]
  We derive the expression of the joint Q value function of LVD and MVD. To ensure optimal consistency, the optimal node is required to be the unique STN. Our method outperforms state-of-the-art baselines in experiments on various benchmarks.
  arXiv  Detail & Related papers  (2022-11-22T08:14:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.