Enhancing Value Alignment of LLMs with Multi-agent system and Combinatorial Fusion

• URL: http://arxiv.org/abs/2603.11126v1
• Date: Wed, 11 Mar 2026 14:57:51 GMT
• Title: Enhancing Value Alignment of LLMs with Multi-agent system and Combinatorial Fusion
• Abstract summary: We propose a framework that operationalizes multi-agent fusion alignment.<n>It mitigates conflicts and redundancies across multiple agents, producing responses that better reflect human values.
• Score: 5.97790552623697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aligning large language models (LLMs) with human values is a central challenge for ensuring trustworthy and safe deployment. While existing methods such as Reinforcement Learning from Human Feedback (RLHF) and its variants have improved alignment, they often rely on a single evaluator or narrowly defined reward signals, limiting their ability to capture ethical pluralism. In this work, we propose the Value Alignment System using Combinatorial Fusion Analysis (VAS-CFA), a framework that operationalizes multi-agent fusion alignment. It instantiates multiple moral agents, each fine-tuned to represent a distinct normative perspective, and fuses their outputs using CFA with both rank- and score-based aggregation. This design leverages cognitive diversity, between agents, to mitigate conflicts and redundancies across multiple agents, producing responses that better reflect human values. Empirical evaluation demonstrates that VAS-CFA outperforms both single agent baselines and prior aggregation approaches on standard metrics, showing that multi-agent fusion provides a robust and effective mechanism for advancing value alignment in LLMs.

Related papers

• Contextual Value Alignment via Multilayer Combinatorial Fusion [5.97790552623697]
  We propose a framework that utilizes multilayer fusion for contextual value alignment (MCF-CVA)<n>At the first layer of the framework, it instantiates multiple moral agents, each fine-tuned to represent a distinctive value.<n>The MCF-CVA framework leverages cognitive diversity between agents to mitigate conflicts and redundancies across multiple agents, producing responses that better reflect contextual human values.
  arXiv  Detail & Related papers  (2026-08-07T16:22:33Z)
• Adaptive Confidence-weighted Expansion for Trustworthy Multi-Omics Multimodal Fusion [0.8749675983608171]
  We introduce Adaptive Confidence-weighted Expansion (ACE) to enhance the trustworthiness of multimodal fusion models.<n>ACE significantly outperforms existing state-of-the-art algorithms in both classification performance and confidence calibration.
  arXiv  Detail & Related papers  (2026-07-22T21:42:16Z)
• Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning [13.30869366778628]
  CRM is a framework that replaces a single black-box reward model with a coordinated team of specialist evaluators.<n>To support training and assessment, we introduce rewardBench, a benchmark and training suite aligned with the collaborative structure of CRM.
  arXiv  Detail & Related papers  (2025-11-20T10:12:34Z)
• Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation [23.41040153806061]
  We propose a novel framework called Multi-Value Alignment (MVA)<n>It mitigates alignment caused by parameter interference among diverse human values by minimizing their mutual information.<n>MVA consistently outperforms existing baselines in aligning LLMs with multiple human values.
  arXiv  Detail & Related papers  (2025-11-15T13:33:26Z)
• CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards [80.78748457530718]
  Self-evolution is a central research topic in enabling large language model (LLM)-based agents to continually improve their capabilities after pretraining.<n>We introduce Co-Evolving Multi-Agent Systems (CoMAS), a novel framework that enables agents to improve autonomously by learning from inter-agent interactions.
  arXiv  Detail & Related papers  (2025-10-09T17:50:26Z)
• Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning [49.31650627835956]
  Partial agent failure becomes inevitable when systems scale up, making it crucial to identify the subset of agents whose compromise would most severely degrade overall performance.<n>In this paper, we study this Vulnerable Agent Identification (VAI) problem in large-scale multi-agent reinforcement learning (MARL)<n> Experiments show our method effectively identifies more vulnerable agents in large-scale MARL and the rule-based system, fooling system into worse failures, and learning a value function that reveals the vulnerability of each agent.
  arXiv  Detail & Related papers  (2025-09-18T16:03:50Z)
• Offline Multi-agent Reinforcement Learning via Score Decomposition [51.23590397383217]
  offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts.<n>This work is the first work to explicitly address the distributional gap between offline and online MARL.
  arXiv  Detail & Related papers  (2025-05-09T11:42:31Z)
• MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncovered [2.8692611791027893]
  We present MALIBU, a novel benchmark developed to assess the degree to which multi-agent systems implicitly reinforce social biases and stereotypes.<n>Our study quantifies biases in LLM-generated outputs, revealing that bias mitigation may favor marginalized personas over true neutrality.
  arXiv  Detail & Related papers  (2025-04-10T19:16:40Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality [52.49062565901046]
  Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models with human values.<n>Existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences.<n>We introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations.
  arXiv  Detail & Related papers  (2025-03-10T09:52:42Z)
• SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learning [23.032729815716813]
  High complexity of real-world environments exacerbates the credit assignment problem.<n>The variability of agent populations in large-scale scenarios necessitates scalable decision-making mechanisms.<n>We propose a novel framework: Sequential rollout with Sequential value estimation (SrSv)
  arXiv  Detail & Related papers  (2025-03-03T12:17:18Z)
• Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration [60.95748658638956]
  This paper introduces the Multi-Label Confidence task, aiming to provide well-calibrated confidence scores in multi-label scenarios. Existing single-label calibration methods fail to account for category correlations, which are crucial for addressing semantic confusion. We propose the Dynamic Correlation Learning and Regularization algorithm, which leverages multi-grained semantic correlations to better model semantic confusion.
  arXiv  Detail & Related papers  (2024-07-09T13:26:21Z)
• Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment [103.05005690990271]
  Mixture of insighTful Experts (MoTE) is a novel framework that combines reasoning chains and expert mixtures to improve self-alignments.<n>MoTE significantly improves model safety, jailbreak resistance, and over-refusal capabilities, achieving performance comparable to OpenAI's state-of-the-art o1 model.
  arXiv  Detail & Related papers  (2024-05-01T15:06:05Z)
• FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC) It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2020-03-14T21:29:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.