LLM-Guided Future Hypotheses for Horizon-Aware Exploration in Multi-Step Robot Manipulation

• URL: http://arxiv.org/abs/2605.29864v1
• Date: Thu, 28 May 2026 12:49:37 GMT
• Title: LLM-Guided Future Hypotheses for Horizon-Aware Exploration in Multi-Step Robot Manipulation
• Abstract summary: Multi-step robot manipulation requires acting under uncertainty about how the scene will evolve.<n>We study whether short-horizon, task-consistent future videos can provide useful structured priors for control and reinforcement-learning fine-tuning.
• Score: 5.637033593506126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-step robot manipulation requires acting under uncertainty about how the scene will evolve, making exploration and policy adaptation challenging. We study whether short-horizon, task-consistent future videos can provide useful structured priors for control and reinforcement-learning fine-tuning. We formalize this idea through Future-Experience Conditioning (FEC), a simple interface that conditions closed-loop policies on a latent representation of a short future video. In our simulation setup, future clips are generated in three stages, an LLM reasoner operating over a task ontology initialized from the current scene state, a robot-free digital-twin rollout of the intended object motion, and a mask-free video diffusion model that synthesizes a robot-consistent future clip without requiring segmentation at inference. We instantiate this future-conditioning interface primarily with BC and BC+RL, and compare against a future-conditioned Streaming Flow Policy (SFP) baseline on RoboCasa and CALVIN under NoFuture, GTFuture, GenFuture, and WrongFuture. Generated futures improve performance over no-future conditioning, while mismatched futures degrade it, and our BC+RL instantiation achieves the strongest overall results. An average BC+RL learning-curve analysis across 8 CALVIN tasks further shows that GTFuture improves fastest, GenFuture improves earlier and to a higher level than NoFuture, and WrongFuture remains at zero throughout training. These results suggest that short-horizon future videos can serve as useful structured priors for exploration and policy adaptation under imperfect future predictions. https://enact2026.github.io/

Related papers

• RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures [51.85284871989367]
  We propose RoboTALES, a framework that learns task-aligned simulated futures and uses them to train robot policies.<n>Our approach introduces two key innovations: (1) a hierarchical LLM-based planner that breaks complex tasks into a sequence of subgoals to guide the model's imagination; and (2) a VLM-based critic that evaluates these imagined'' futures and uses reward-based feedback to keep the model's internal representations focused on the goal.
  arXiv  Detail & Related papers  (2026-07-07T08:58:29Z)
• InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization [81.9611022323101]
  We present InternVLA-A1.5, which builds the policy on a native VLM backbone that keeps training on VQA and subtask prediction, and attaches a lightweight unified expert for continuous action generation.<n>Future prediction is recast as a latent-querying problem, where a small set of learnable foresight tokens condenses the task-relevant future into a compact latent code under the supervision of a frozen pretrained video generation model.<n>Pretrained on 1.2M robot episodes and 3M multimodal samples, InternVLA-A1.5 achieves the best overall results on all six simulation benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T12:25:30Z)
• DexFuture: Hierarchical Future-State Visuomotor Targeting for Bimanual Dexterous Tool Use [10.996987671079046]
  Bimanual dexterous tool use remains challenging for robots due to high-dimensional hand configurations and complex hand-tool-object dynamics and contact.<n>We propose DexFuture, a hierarchical system that couples a high-level Future-State Visuomotor Target Predictor with a low-level Target-Conditioned Structured Dexterous Policy.<n>On OakInk2 bimanual tool-use tasks, DexFuture achieves 90% of the privileged-oracle performance, compared to 7% for a no-reference policy.
  arXiv  Detail & Related papers  (2026-06-04T04:37:23Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• GigaWorld-Policy: An Efficient Action-Centered World--Action Model [50.107640832046464]
  GigaWorld-Policy is an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation.<n> Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus.
  arXiv  Detail & Related papers  (2026-03-18T00:52:02Z)
• Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations [19.45821593625599]
  Video diffusion models (VDMs) demonstrate the ability to predict future frames and showcase a strong understanding of physical world.<n>We propose the Video Prediction Policy (VPP), which learns implicit inverse dynamics model conditioned on predicted future representations inside VDMs.<n>VPP achieves a 18.6% relative improvement on the Calvin ABC-D generalization benchmark compared to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2024-12-19T12:48:40Z)
• GePSAn: Generative Procedure Step Anticipation in Cooking Videos [22.908511747910197]
  We study the problem of future step anticipation in procedural videos. Given a video of an ongoing procedural activity, we predict a plausible next procedure step described in rich natural language. We show that our model captures diversity in the next step prediction and generates multiple plausible future predictions.
  arXiv  Detail & Related papers  (2023-10-12T13:20:17Z)
• Policy Gradients Incorporating the Future [66.20567145291342]
  We introduce a method that allows an agent to "look into the future" without explicitly predicting it. We propose to allow an agent, during its training on past experience, to observe what emphactually happened in the future at that time. This gives our agent the opportunity to utilize rich and useful information about the future trajectory dynamics in addition to the present.
  arXiv  Detail & Related papers  (2021-08-04T14:57:11Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)
• Learning to Anticipate Egocentric Actions by Imagination [60.21323541219304]
  We study the egocentric action anticipation task, which predicts future action seconds before it is performed for egocentric videos. Our method significantly outperforms previous methods on both the seen test set and the unseen test set of the EPIC Kitchens Action Anticipation Challenge.
  arXiv  Detail & Related papers  (2021-01-13T08:04:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.