MOA: Multi-Objective Alignment for Role-Playing Agents

• URL: http://arxiv.org/abs/2512.09756v1
• Date: Wed, 10 Dec 2025 15:35:11 GMT
• Title: MOA: Multi-Objective Alignment for Role-Playing Agents
• Authors: Chonghua Liao, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li,
• Abstract summary: We present MOA, a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs.<n>To address the issues of model output diversity and quality, we have also employed thought-augmented rollout with off-policy guidance.<n>This demonstrates the great potential of MOA in building RPAs that can simultaneously meet the demands of role knowledge, persona style, diverse scenarios, and complex multi-turn conversations.
• Score: 68.67837397211903
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Role-playing agents (RPAs) must simultaneously master many conflicting skills -- following multi-turn instructions, exhibiting domain knowledge, and adopting a consistent linguistic style. Existing work either relies on supervised fine-tuning (SFT) that over-fits surface cues and yields low diversity, or applies reinforcement learning (RL) that fails to learn multiple dimensions for comprehensive RPA optimization. We present MOA (Multi-Objective Alignment), a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs. MOA introduces a novel multi-objective optimization strategy that trains simultaneously on multiple fine-grained rubrics to boost optimization performance. Besides, to address the issues of model output diversity and quality, we have also employed thought-augmented rollout with off-policy guidance. Extensive experiments on challenging benchmarks such as PersonaGym and RoleMRC show that MOA enables an 8B model to match or even outperform strong baselines such as GPT-4o and Claude across numerous dimensions. This demonstrates the great potential of MOA in building RPAs that can simultaneously meet the demands of role knowledge, persona style, diverse scenarios, and complex multi-turn conversations.

Related papers

• MARTI-MARS$^2$: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation [64.2621682259008]
  Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2)<n>We propose a Multi-Agent Reinforced Training and Inference Framework with Self-Search Scaling (MARTI-MARS2) to integrate policy learning with multi-agent tree search.<n>We show that MARTI-MARS2 achieves 77.7%, outperforming strong baselines like GPT-5.1 on challenging code generation benchmarks.
  arXiv  Detail & Related papers  (2026-02-08T07:28:44Z)
• MMhops-R1: Multimodal Multi-hop Reasoning [89.68086555694084]
  We introduce MMhops, a novel benchmark designed to evaluate and foster multi-modal multi-hop reasoning.<n> MMhops dataset comprises two challenging task formats, Bridging and Comparison.<n>We propose MMhops-R1, a novel multi-modal Retrieval-Augmented Generation framework for dynamic reasoning.
  arXiv  Detail & Related papers  (2025-12-15T17:29:02Z)
• MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning [18.0412262027514]
  We propose a novel Mixture of Low-Rank Experts (MoRE) for multi-task.<n>Instead of using an individual LoRA for each task, we align different ranks of LoRA module with different tasks.<n>We also design a novel adaptive rank selector to select the appropriate expert for each task.
  arXiv  Detail & Related papers  (2025-05-28T12:32:09Z)
• MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models [34.138699712315]
  This paper introduces a novel vision--action (VLA) model, mixture of robotic experts (MoRE) for quadruped robots.<n>MoRE integrates multiple low-rank adaptation modules as distinct experts within a dense multi-modal large language model.<n>Experiments demonstrate that MoRE outperforms all baselines across six different skills and exhibits superior generalization capabilities in out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-03-11T03:13:45Z)
• In Search for Architectures and Loss Functions in Multi-Objective Reinforcement Learning [0.6650227510403052]
  Multi-objective reinforcement learning (MORL) is essential for addressing the intricacies of real-world RL problems. MORL is challenging due to unstable learning dynamics with deep learning-based function approximators. Our work empirically explores model-free policy learning loss functions and the impact of different architectural choices.
  arXiv  Detail & Related papers  (2024-07-23T19:17:47Z)
• Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning [51.52387511006586]
  We propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies. HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios.
  arXiv  Detail & Related papers  (2024-06-12T08:48:06Z)
• T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning [31.276142111455847]
  Large language models (LLMs) encounter significant adaptation challenges in diverse multitask finetuning.<n>We design a novel framework, mixunderlinetextbfTureunderlinetextbf-of-underlinetextbfRank-onunderlinetextbfE-eunderlinetextbfXper ts (textttT-REX)<n>Rank-1 experts enable a mix-and-match mechanism to quadratically expand the vector subspace of experts with linear parameter overheads, achieving approximate error reduction with optimal
  arXiv  Detail & Related papers  (2024-04-13T12:14:58Z)
• Demonstration Guided Multi-Objective Reinforcement Learning [2.9845592719739127]
  We introduce demonstration-guided multi-objective reinforcement learning (DG-MORL) This novel approach utilizes prior demonstrations, aligns them with user preferences via corner weight support, and incorporates a self-evolving mechanism to refine suboptimal demonstrations. Our empirical studies demonstrate DG-MORL's superiority over existing MORL algorithms, establishing its robustness and efficacy.
  arXiv  Detail & Related papers  (2024-04-05T10:19:04Z)
• Multimodal Instruction Tuning with Conditional Mixture of LoRA [51.58020580970644]
  This paper introduces a novel approach that integrates multimodal instruction tuning with Low-Rank Adaption (LoRA)<n>It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance.<n> Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks.
  arXiv  Detail & Related papers  (2024-02-24T20:15:31Z)
• Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE [83.00018517368973]
  Large Language Models (LLMs) can extend their zero-shot capabilities to multimodal learning through instruction tuning. negative conflicts and interference may have a worse impact on performance. We combine the well-known Mixture-of-Experts (MoE) and one of the representative PEFT techniques, i.e., LoRA, designing a novel LLM-based decoder, called LoRA-MoE, for multimodal learning.
  arXiv  Detail & Related papers  (2023-11-05T15:48:29Z)
• Multi-objective Pointer Network for Combinatorial Optimization [10.286195356515355]
  Multi-objective optimization problems (MOCOPs) exist in various real applications. Deep reinforcement learning (DRL) methods have been proposed to generate approximate optimal solutions to the optimization problems. This study proposes a single-model deep reinforcement learning framework, called multi-objective Pointer Network (MOPN)
  arXiv  Detail & Related papers  (2022-04-25T14:02:34Z)
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers [108.92194081987967]
  We make the first attempt to explore a universal multi-agent reinforcement learning pipeline, designing one single architecture to fit tasks. Unlike previous RNN-based models, we utilize a transformer-based model to generate a flexible policy. The proposed model, named as Universal Policy Decoupling Transformer (UPDeT), further relaxes the action restriction and makes the multi-agent task's decision process more explainable.
  arXiv  Detail & Related papers  (2021-01-20T07:24:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.