MAPL: Multi-Objective Preference Learning for Robot Locomotion

• URL: http://arxiv.org/abs/2606.25398v1
• Date: Wed, 24 Jun 2026 04:52:04 GMT
• Title: MAPL: Multi-Objective Preference Learning for Robot Locomotion
• Abstract summary: Reward design remains a major bottleneck in reinforcement learning for robot locomotion.<n>We present Multi-Objective AI-Informed Preference Learning (MAPL), a framework that learns locomotion rewards from high-level natural language objectives.
• Score: 3.975297231968465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward design remains a major bottleneck in reinforcement learning for robot locomotion, where successful policies often depend on carefully tuned, task-specific reward functions. Preference-based reinforcement learning offers an alternative, but existing LLM-based methods typically ask for a single overall judgment between behaviors, making it difficult to capture the multiple competing objectives that underlie high-quality locomotion. We present Multi-Objective AI-Informed Preference Learning (MAPL), a framework that learns locomotion rewards from high-level natural language objectives rather than manually engineered reward equations. MAPL prompts a large language model to compare trajectories independently along semantically meaningful criteria, using generic language descriptions that are terrain-invariant and require little domain expertise. These objective-wise preferences are used to train a multi-head preference scoring model, whose outputs are aggregated to form a scalar reward for policy optimization. Across four quadruped locomotion environments, MAPL trains policies using only LLM-generated preferences and achieves performance comparable to or better than expert-designed rewards, while eliminating task-specific reward engineering.

Related papers

• LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback [4.210862582629798]
  We present LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback.<n>It is a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies.<n>Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives.
  arXiv  Detail & Related papers  (2026-07-31T15:50:29Z)
• MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models [47.91243135663066]
  We introduce MATO, a training-free framework for Multi-objective personalized alignment with Test-time Optimization.<n> MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding.
  arXiv  Detail & Related papers  (2026-05-25T01:57:22Z)
• One Model for All: Multi-Objective Controllable Language Models [65.4626816393381]
  We introduce Multi-Objective Control (MOC) to train a single language model as a preference-conditioned policy network.<n>Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditioned policy network.<n>Extensive experiments demonstrate the advantages of MOC over baselines in three aspects.
  arXiv  Detail & Related papers  (2026-04-06T07:48:32Z)
• PCHC: Enabling Preference Conditioned Humanoid Control via Multi-Objective Reinforcement Learning [50.63196995993855]
  We propose a novel framework leveraging Multi-Objective Reinforcement Learning (MORL) to achieve Preference-Conditioned Humanoid Control (PCHC)<n>Our framework enables a single, preference-conditioned policy to exhibit a wide spectrum of diverse behaviors.
  arXiv  Detail & Related papers  (2026-03-25T07:55:37Z)
• Preference-based Multi-Objective Reinforcement Learning [5.031225669460861]
  This paper introduces preference-based MORL (Pb-MORL), which formalizes the integration of preferences into the MORL framework.<n>To guide policy optimization using preferences, our method constructs a multi-objective reward model that aligns with the given preferences.<n>Experiments in benchmark multi-objective tasks, a multi-energy management task, and an autonomous driving task on a multi-line highway show that our method performs competitively.
  arXiv  Detail & Related papers  (2025-07-18T16:43:04Z)
• LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning [13.035613181550941]
  Large Language Model-Assisted Preference Prediction (LAPP) is a novel framework for robot learning.<n>LAPP enables efficient, customizable, and expressive behavior acquisition with minimum human effort.<n>We show that LAPP achieves efficient learning, higher final performance, faster adaptation, and precise control of high-level behaviors.
  arXiv  Detail & Related papers  (2025-04-21T22:46:29Z)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks.<n>However, they still struggle with problems requiring multi-step decision-making and environmental feedback.<n>We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Goal-Conditioned Supervised Learning for Multi-Objective Recommendation [8.593384839118658]
  Multi-objective learning endeavors to concurrently optimize multiple objectives using a single model.<n>This paper introduces a Multi-Objective Goal-Conditioned Supervised Learning framework for automatically learning to achieve multiple objectives from offline sequential data.
  arXiv  Detail & Related papers  (2024-12-12T03:47:40Z)
• Self-Exploring Language Models: Active Preference Elicitation for Online Alignment [88.56809269990625]
  We propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, Self-Exploring Language Models (SELM) significantly boosts the performance on instruction-following benchmarks.
  arXiv  Detail & Related papers  (2024-05-29T17:59:07Z)
• Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models [41.38520841504846]
  Large language models (LLMs) can provide automated preference feedback solely from image inputs to guide decision-making. In this study, we train a multimodal LLM, termed CriticGPT, capable of understanding trajectory videos in robot manipulation tasks. Experimental evaluation of the algorithm's preference accuracy demonstrates its effective generalization ability to new tasks. Performance on Meta-World tasks reveals that CriticGPT's reward model efficiently guides policy learning, surpassing rewards based on state-of-the-art pre-trained representation models.
  arXiv  Detail & Related papers  (2024-02-22T03:14:03Z)
• Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization [76.09576643028362]
  We present Multi-Objective Direct Preference Optimization (MODPO) for multiple alignment objectives. MODPO folds language modeling directly into reward modeling, training language models as implicit collective reward models. It theoretically yields the same optimal solutions as MORLHF but is practically more stable and efficient.
  arXiv  Detail & Related papers  (2023-10-05T17:35:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.