MCTS-EP: Empowering Embodied Planning with Online Preference Optimization

• URL: http://arxiv.org/abs/2509.17116v1
• Date: Sun, 21 Sep 2025 15:17:44 GMT
• Title: MCTS-EP: Empowering Embodied Planning with Online Preference Optimization
• Abstract summary: This paper introduces MCTS-EP, an online learning framework that combines large language models (LLM) with Monte Carlo Tree Search (MCTS) for training embodied agents.<n>We theoretically prove that MCTS-EP achieves better performance bounds than conventional on-policy algorithms when the loss function is strongly convex.
• Score: 42.11570460915765
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces MCTS-EP, an online learning framework that combines large language models (LLM) with Monte Carlo Tree Search (MCTS) for training embodied agents. MCTS-EP integrates three key components: MCTS-guided exploration for preference data collection, efficient multi-modal reasoning mechanism, and iterative training pipeline based on preference optimization. We theoretically prove that MCTS-EP achieves better performance bounds than conventional on-policy algorithms when the loss function is strongly convex, and demonstrate that it can be formulated as a search-enhanced variant of GAIL. MCTS-EP achieves state-of-the-art performace across serval benchmarks. In ALFWorld, it achieves 92% and 87% success rates for textual and visual tasks. In WebShop, it reaches an average reward of 0.81. MTCS-EP also reduces average interaction steps from from 18.7/19.5 to 10.2/9.9 steps in visual ALFWorld.Code available at: https://github.com/xuhang-2/Embodied-Agent-Planning

Related papers

• OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation [39.548179971747906]
  Domain-specific finetuning is essential for dense retrievers, yet not all training pairs contribute equally to the learning process.<n>We introduce OPERA, a data pruning framework that exploits this heterogeneity to improve both the effectiveness and efficiency of retrieval model adaptation.
  arXiv  Detail & Related papers  (2026-03-17T23:11:45Z)
• Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS [62.22644307952087]
  We introduce AIRL-S, the first natural unification of RL-based and search-based TTS.<n>We leverage adversarial inverse reinforcement learning (AIRL) combined with group relative policy optimization (GRPO) to learn a dense, dynamic PRM directly from correct reasoning traces.<n>Our results show that our unified approach improves performance by 9 % on average over the base model, matching GPT-4o.
  arXiv  Detail & Related papers  (2025-08-19T23:41:15Z)
• UniECS: Unified Multimodal E-Commerce Search Framework with Gated Cross-modal Fusion [20.13803245640432]
  Current e-commerce multimodal retrieval systems face two key limitations.<n>They optimize for specific tasks with fixed modality pairings, and lack comprehensive benchmarks for evaluating unified retrieval approaches.<n>We introduce UniECS, a unified multimodal e-commerce search framework that handles all retrieval scenarios across image, text, and their combinations.
  arXiv  Detail & Related papers  (2025-08-19T14:06:13Z)
• A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning [0.40964539027092906]
  Supervised Fine-Tuning and Reinforcement Learning are the dominant training paradigms.<n>This paper introduces a practical and effective training recipe that strategically integrates extended SFT with RL from online inference.<n>Our experiments reveal that extending SFT for as many as 10 epochs is crucial for performance breakthroughs.<n>This work provides the community with a battle-tested blueprint for developing state-of-the-art mathematical reasoners.
  arXiv  Detail & Related papers  (2025-07-11T02:26:01Z)
• Doubly Robust Monte Carlo Tree Search [0.0]
  We present Doubly Robust Monte Carlo Tree Search (DR-MCTS), a novel algorithm that integrates Doubly Robust (DR) off-policy estimation into Monte Carlo Tree Search (MCTS)<n>Our approach combines MCTS rollouts with DR estimation, offering theoretical guarantees of unbiasedness and variance reduction under specified conditions.<n> Empirical evaluations in Tic-Tac-Toe and the partially observable VirtualHome environment demonstrate DR-MCTS's superior performance over standard MCTS.
  arXiv  Detail & Related papers  (2025-02-01T19:32:46Z)
• Kimi k1.5: Scaling Reinforcement Learning with LLMs [84.95584393629998]
  We report on the training practice of Kimi k1.5, our latest multi-modal language model trained with reinforcement learning.<n>Long context scaling and improved policy optimization methods are key ingredients of our approach.<n>Our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities.
  arXiv  Detail & Related papers  (2025-01-22T02:48:14Z)
• EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference [49.94169109038806]
  This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that surpasses the existing parallelism schemes.<n>Our results demonstrate at most 52.4% improvement in prefill throughput compared to existing parallel inference methods.
  arXiv  Detail & Related papers  (2024-10-16T05:17:49Z)
• Optimized Monte Carlo Tree Search for Enhanced Decision Making in the FrozenLake Environment [0.0]
  Monte Carlo Tree Search (MCTS) is a powerful algorithm for solving complex decision-making problems. This paper presents an optimized MCTS implementation applied to the FrozenLake environment, a classic reinforcement learning task.
  arXiv  Detail & Related papers  (2024-09-25T05:04:53Z)
• Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning [55.96599486604344]
  We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process. We use Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data.
  arXiv  Detail & Related papers  (2024-05-01T11:10:24Z)
• Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration [87.53543137162488]
  We propose an easy-to-implement online reinforcement learning (online RL) framework called textttMEX. textttMEX integrates estimation and planning components while balancing exploration exploitation automatically. It can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards.
  arXiv  Detail & Related papers  (2023-05-29T17:25:26Z)
• Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval [94.73459295405507]
  Smooth-AP is a plug-and-play objective function that allows for end-to-end training of deep networks. We apply Smooth-AP to standard retrieval benchmarks: Stanford Online products and VehicleID. We also evaluate on larger-scale datasets: INaturalist for fine-grained category retrieval, VGGFace2 and IJB-C for face retrieval.
  arXiv  Detail & Related papers  (2020-07-23T17:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.