DexFuture: Hierarchical Future-State Visuomotor Targeting for Bimanual Dexterous Tool Use

• URL: http://arxiv.org/abs/2606.05699v1
• Date: Thu, 04 Jun 2026 04:37:23 GMT
• Title: DexFuture: Hierarchical Future-State Visuomotor Targeting for Bimanual Dexterous Tool Use
• Abstract summary: Bimanual dexterous tool use remains challenging for robots due to high-dimensional hand configurations and complex hand-tool-object dynamics and contact.<n>We propose DexFuture, a hierarchical system that couples a high-level Future-State Visuomotor Target Predictor with a low-level Target-Conditioned Structured Dexterous Policy.<n>On OakInk2 bimanual tool-use tasks, DexFuture achieves 90% of the privileged-oracle performance, compared to 7% for a no-reference policy.
• Score: 10.996987671079046
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Bimanual dexterous tool use remains challenging for robots due to high-dimensional hand configurations and complex hand-tool-object dynamics and contact. Most existing control policies depend on future configuration references provided from demonstrations, while future action-conditioned world models require slow online planning over high-dimensional action sequences. A significant challenge is generating a dynamically consistent future reference trajectory without relying on privileged states from demonstrations or slow counterfactual planning. We propose DexFuture, a hierarchical system that couples a high-level Future-State Visuomotor Target Predictor with a low-level Target-Conditioned Structured Dexterous Policy. Conditioned on egocentric RGB, proprioceptive and geometric history, the high-level predictor constructs structured hand-tool-object visuomotor embeddings and uses a horizon-conditioned transformer to generate a multi-step future target trajectory. Then, the low-level policy tracks them with a target-conditioned per-link transformer. This hierarchy decouples coarse future reference generation from fine-grained action control, and slow long-horizon semantic prediction from high-frequency execution. On OakInk2 bimanual tool-use tasks, DexFuture achieves 90% of the privileged-oracle performance, compared to 7% for a no-reference policy. DexFuture operates at 60 Hz, approximately 250 times faster than DexWM-style Cross-Entropy Method (CEM) planning with a future action-conditioned world model.

Related papers

• WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression [24.010807495676605]
  Generative world models provide robots with predictive models of how the world evolves under interaction.<n>We introduce WALL-SS, a world model that generates visual futures through Scale-wise autoregressive Scaling.<n>We show that WALL-SS improves action following and trajectory accuracy, supports coherent minute-long streaming rollout under bounded memory, and consistently benefits from on-policy alignment.
  arXiv  Detail & Related papers  (2026-08-26T17:57:12Z)
• Geometric Action Model for Robot Policy Learning [68.6657929619782]
  Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world.<n>Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models.<n>We propose the Geometric Action Model (GAM), a language-conditioned manipulation policy.
  arXiv  Detail & Related papers  (2026-06-15T17:58:03Z)
• LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies [31.88655699486955]
  We present LaWAM, a latent-action-conditioned Latent World Model (LaWM)<n>LaWAM exposes predictive dynamics to robot policies through compact latent visual subgoals instead of reconstructed future video.<n>LaWAM runs in 187 ms per action-chunk prediction and achieves up to 24x lower wall-clock latency than pixel-space WAMs.
  arXiv  Detail & Related papers  (2026-06-14T12:06:58Z)
• LLM-Guided Future Hypotheses for Horizon-Aware Exploration in Multi-Step Robot Manipulation [5.637033593506126]
  Multi-step robot manipulation requires acting under uncertainty about how the scene will evolve.<n>We study whether short-horizon, task-consistent future videos can provide useful structured priors for control and reinforcement-learning fine-tuning.
  arXiv  Detail & Related papers  (2026-05-28T12:49:37Z)
• GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation [54.671815855499034]
  Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors to action generation.<n>Standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution.<n>We propose textbfGaussianDream, a feed-forward 3D Gaussian world-model plug-in.
  arXiv  Detail & Related papers  (2026-05-20T05:51:30Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation [57.07494675832939]
  Most existing vision-language-action (VLA) models for robotic manipulation lack progress awareness.<n>In this work, we investigate the estimation and integration of task progress, proposing a novel model named textbf vla<n>Experiments on the CALVIN and LIBERO benchmarks, alongside real-world robot deployment, consistently demonstrate substantial improvements in success rates.
  arXiv  Detail & Related papers  (2026-03-29T12:38:11Z)
• RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Reset [48.645870795753105]
  We introduce Robust Autonomous Data Acquisition for Robotics (RADAR)<n>RADAR is a fully autonomous, closed-loop data generation engine that completely removes human intervention from the collection cycle.<n>In simulation, our framework achieves up to 90% success rates on complex, long-horizon tasks.
  arXiv  Detail & Related papers  (2026-03-12T11:18:52Z)
• UltraDexGrasp: Learning Universal Dexterous Grasping for Bimanual Robots with Synthetic Data [70.96218513410454]
  We introduce UltraDexGrasp, a framework for universal dexterous grasping with bimanual robots.<n>The proposed data-generation pipeline integrates optimization-based grasp synthesis with planning-based demonstration generation.<n>Based on UltraDexGrasp-20M, we develop a simple yet effective grasp policy that takes point clouds as input, aggregates scene features via unidirectional attention, and predicts control commands.
  arXiv  Detail & Related papers  (2026-03-05T15:54:21Z)
• GR-Dexter Technical Report [32.12634986073996]
  Vision-language-action (VLA) models have enabled language-conditioned, long-horizon robot manipulation.<n>We present GR-Dexter, a holistic hardware-model-data framework for VLA-based generalist manipulation on a bimanual dexterous-hand robot.
  arXiv  Detail & Related papers  (2025-12-30T13:22:16Z)
• Ctrl-World: A Controllable Generative World Model for Robot Manipulation [53.71061464925014]
  Generalist robot policies can perform a wide range of manipulation skills.<n> evaluating and improving their ability with unfamiliar objects and instructions remains a significant challenge.<n>World models offer a promising, scalable alternative by enabling policies to rollout within imagination space.
  arXiv  Detail & Related papers  (2025-10-11T09:13:10Z)
• Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data [37.176411554794214]
  Reasoning about human motion is an important prerequisite to safe and socially-aware robotic navigation. We present Trajectron++, a modular, graph-structured recurrent model that forecasts the trajectories of a general number of diverse agents. We demonstrate its performance on several challenging real-world trajectory forecasting datasets.
  arXiv  Detail & Related papers  (2020-01-09T16:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.