Discovering Diverse Planning Policies for Multimodal Embodied Agents with Quality-Diversity Optimization

• URL: http://arxiv.org/abs/2608.08523v1
• Date: Sun, 09 Aug 2026 06:41:53 GMT
• Title: Discovering Diverse Planning Policies for Multimodal Embodied Agents with Quality-Diversity Optimization
• Abstract summary: We propose a Quality-Diversity framework for discovering diverse planning policies for multimodal embodied agents.<n>The proposed method treats planning-policy templates as evolvable individuals and organizes them into a behavior-indexed archive.<n>In the offline stage, rollout trajectories are summarized into structured success and failure experiences, which guide policy variation.<n>In the online stage, the system rolls back to the latest checkpoint and switches to a behaviorally distinct archive policy to resume execution.
• Score: 14.306388130938236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal embodied agents are increasingly required to solve long-horizon tasks by integrating visual observations, textual goals, and interaction history into closed-loop decision making. However, state-of-the-art large-model-based planners often rely on a single dominant planning style during execution. Once this execution mode becomes ineffective, the agent may remain stalled for many steps, repeatedly interacting with the environment without making meaningful progress. We address this limitation by proposing a Quality-Diversity (QD) framework for discovering diverse planning policies for multimodal embodied agents. The proposed method treats planning-policy templates as evolvable individuals and organizes them into a behavior-indexed archive rather than collapsing search to a single prompt style. In the offline stage, rollout trajectories are summarized into structured success and failure experiences, which guide policy variation through recombination and experience-guided mutation. The resulting policies are mapped into a behavior space defined by interaction intensity and goal-directedness, and the highest-quality policy in each niche is retained in the archive. In the online stage, the agent executes one policy at a time while monitoring task progress. When persistent stall is detected, the system rolls back to the latest checkpoint and switches to a behaviorally distinct archive policy to resume execution. Experiments on the ThreeDWorld transport benchmark show that the proposed framework improves both task success and interaction efficiency over representative baseline planners. These results suggest that discovering diverse policy repertoires is an effective way to support adaptive multimodal planning and online failure recovery.

Related papers

• OpenLoopEvolve: A Verifiable Self-Evolution Framework for Loop Policies in Long-Horizon Complex Tasks [17.773038952370865]
  This paper presents OpenLoopEvolve (OLE), a self-evolution framework centered on the Loop Policy.<n>OLE represents an agent's observation, planning, memory, action, verification, recovery, stopping, and budget control as portable policy assets with versions and lineages.<n>Online mode triggers candidate generation based on feedback from continuous operation, whereas the offline mode searches for candidate policies from archived traces and failure evidence.
  arXiv  Detail & Related papers  (2026-08-10T09:57:26Z)
• Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies [7.282730307270664]
  We address the problem of fine-tuning pre-trained generative policies with reinforcement learning (RL)<n>Existing methods for RL fine-tuning of generative policies collapse diverse behaviors into a single reward-maximizing mode.<n>We propose an unsupervised mode discovery framework that uncovers latent behavioral modes within generative policies.
  arXiv  Detail & Related papers  (2026-05-12T01:19:09Z)
• HiPolicy: Hierarchical Multi-Frequency Action Chunking for Policy Learning [17.3424491065084]
  HiPolicy is a hierarchical multi-frequency action chunking framework that predicts action sequences at different frequencies to capture both coarse high-level plans and precise reactive motions.<n> Experiments on diverse simulated benchmarks and real-world manipulation tasks show that HiPolicy can be seamlessly integrated into existing 2D and 3D generative policies.
  arXiv  Detail & Related papers  (2026-04-07T16:47:38Z)
• AgentSwing: Adaptive Parallel Context Management Routing for Long-Horizon Web Agents [115.18197165538943]
  Large language models (LLMs) evolve into autonomous agents for long-horizon information-seeking.<n>Existing context management methods typically commit to a single fixed strategy throughout the entire trajectory.<n>We introduce a probabilistic framework that characterizes long-horizon success through two complementary dimensions: search efficiency and terminal precision.
  arXiv  Detail & Related papers  (2026-03-29T02:59:27Z)
• From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderation [59.27094165576015]
  We propose a novel learning paradigm (UniMod) that transitions from sparse decision-making to dense reasoning traces.<n>By constructing structured trajectories encompassing evidence grounding, modality assessment, risk mapping, policy decision, and response generation, we reformulate monolithic decision tasks into a multi-dimensional boundary learning process.<n>We introduce specialized optimization strategies to decouple task-specific parameters and rebalance training dynamics, effectively resolving interference between diverse objectives in multi-task learning.
  arXiv  Detail & Related papers  (2026-01-28T09:29:40Z)
• Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies [36.44665658496622]
  This work introduces a novel approach for programmatic control policy discovery, called Multimodal Large Language Model-assisted Evolutionary Search (MLES)<n>MLES utilizes multimodal large language models as programmatic policy generators, combining them with evolutionary search to automate policy generation.<n> Experimental results demonstrate that MLES achieves performance comparable to Proximal Policy Optimization (PPO) across two standard control tasks.
  arXiv  Detail & Related papers  (2025-08-07T14:24:03Z)
• Offline Multi-agent Reinforcement Learning via Score Decomposition [51.23590397383217]
  offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts.<n>This work is the first work to explicitly address the distributional gap between offline and online MARL.
  arXiv  Detail & Related papers  (2025-05-09T11:42:31Z)
• Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization [8.877649895977479]
  offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets.<n>In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic.<n>We propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO)
  arXiv  Detail & Related papers  (2024-12-10T16:19:08Z)
• Learning From Good Trajectories in Offline Multi-Agent Reinforcement Learning [98.07495732562654]
  offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets. One agent learned by offline MARL often inherits this random policy, jeopardizing the performance of the entire team. We propose a novel framework called Shared Individual Trajectories (SIT) to address this problem.
  arXiv  Detail & Related papers  (2022-11-28T18:11:26Z)
• Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space [76.46113138484947]
  General-purpose robots require diverse repertoires of behaviors to complete challenging tasks in real-world unstructured environments. To address this issue, goal-conditioned reinforcement learning aims to acquire policies that can reach goals for a wide range of tasks on command. We propose Planning to Practice, a method that makes it practical to train goal-conditioned policies for long-horizon tasks.
  arXiv  Detail & Related papers  (2022-05-17T06:58:17Z)
• Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints [52.58352707495122]
  We present a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination. We validate our results over a wide range of simulations on two distinct domains: multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
  arXiv  Detail & Related papers  (2020-05-27T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.