MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis

• URL: http://arxiv.org/abs/2506.18897v2
• Date: Wed, 20 Aug 2025 07:07:13 GMT
• Title: MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis
• Authors: Xiaowei Chi, Kuangzhi Ge, Jiaming Liu, Siyuan Zhou, Peidong Jia, Zichen He, Yuzhen Liu, Tingguang Li, Lei Han, Sirui Han, Shanghang Zhang, Yike Guo,
• Abstract summary: We propose Manipulate in Dream (MinD), a dual-system world model for real-time, risk-aware planning.<n>MinD uses two asynchronous diffusion processes: a low-frequency visual generator (LoDiff) that predicts future scenes and a high-frequency diffusion policy (HiDiff) that outputs actions.<n>Our key insight is that robotic policies do not require fully denoised frames but can rely on low-resolution latents generated in a single denoising step.<n>MinD achieves a 63% success rate on RL-Bench, 60% on real-world Franka tasks, and operates at 11.3 FPS
• Score: 32.08769443927576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Generation Models (VGMs) have become powerful backbones for Vision-Language-Action (VLA) models, leveraging large-scale pretraining for robust dynamics modeling. However, current methods underutilize their distribution modeling capabilities for predicting future states. Two challenges hinder progress: integrating generative processes into feature learning is both technically and conceptually underdeveloped, and naive frame-by-frame video diffusion is computationally inefficient for real-time robotics. To address these, we propose Manipulate in Dream (MinD), a dual-system world model for real-time, risk-aware planning. MinD uses two asynchronous diffusion processes: a low-frequency visual generator (LoDiff) that predicts future scenes and a high-frequency diffusion policy (HiDiff) that outputs actions. Our key insight is that robotic policies do not require fully denoised frames but can rely on low-resolution latents generated in a single denoising step. To connect early predictions to actions, we introduce DiffMatcher, a video-action alignment module with a novel co-training strategy that synchronizes the two diffusion models. MinD achieves a 63% success rate on RL-Bench, 60% on real-world Franka tasks, and operates at 11.3 FPS, demonstrating the efficiency of single-step latent features for control signals. Furthermore, MinD identifies 74% of potential task failures in advance, providing real-time safety signals for monitoring and intervention. This work establishes a new paradigm for efficient and reliable robotic manipulation using generative world models.

Related papers

• Causal World Modeling for Robot Control [56.31803788587547]
  Video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics.<n>We introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously.<n>We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations.
  arXiv  Detail & Related papers  (2026-01-29T17:07:43Z)
• Future Optical Flow Prediction Improves Robot Control & Video Generation [100.87884718953099]
  We introduce FOFPred, a novel optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture.<n>Our model is trained on web-scale human activity data-a highly scalable but unstructured source.<n> Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred.
  arXiv  Detail & Related papers  (2026-01-15T18:49:48Z)
• Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulation [41.993197533574126]
  Inferix is an inference engine to enable immersive world synthesis through optimized semi-autoregressive decoding processes.<n>Inferix further enhances its offering with interactive video streaming and profiling, enabling real-time interaction and realistic simulation.
  arXiv  Detail & Related papers  (2025-11-25T01:45:04Z)
• Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model [62.889356203346985]
  We propose DUal-STream diffusion (DUST), a world-model augmented VLA framework that handles the modality conflict.<n>DUST achieves up to 6% gains over a standard VLA baseline and implicit world-modeling methods.<n>On real-world tasks with the Franka Research 3, DUST outperforms baselines in success rate by 13%.
  arXiv  Detail & Related papers  (2025-10-31T16:32:12Z)
• dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thought [66.78110237549087]
  Vision-Language-Action (VLA) models are emerging as a next-generation paradigm for robotics.<n>We introduce dVLA, a diffusion-based VLA that unifies visual perception, language reasoning, and robotic control in a single system.
  arXiv  Detail & Related papers  (2025-09-30T02:36:11Z)
• Towards Universal Modal Tracking with Online Dense Temporal Token Learning [66.83607018706519]
  We propose a universal video-level modality-awareness tracking model with online dense temporal token learning.<n>We expand the model's inputs to a video sequence level, aiming to see a richer video context from a near-global perspective.
  arXiv  Detail & Related papers  (2025-07-27T08:47:42Z)
• DIFFUMA: High-Fidelity Spatio-Temporal Video Prediction via Dual-Path Mamba and Diffusion Enhancement [5.333662480077316]
  We release the Chip Dicing Lane dataset (CHDL), the first public temporal image dataset dedicated to the semiconductor wafer dicing process.<n>We propose DIFFUMA, an innovative dual-path prediction architecture specifically designed for such fine-grained dynamics.<n>Experiments demonstrate that DIFFUMA significantly outperforms existing methods, reducing the Mean Squared Error (MSE) by 39% and improving the Similarity (SSIM) from 0.926 to a near-perfect 0.988.
  arXiv  Detail & Related papers  (2025-07-09T10:51:54Z)
• DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge [56.3802428957899]
  We propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling.<n>DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning.<n>Experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks.
  arXiv  Detail & Related papers  (2025-07-06T16:14:29Z)
• Epona: Autoregressive Diffusion World Model for Autonomous Driving [39.389981627403316]
  Existing video diffusion models struggle with flexible-length, long-horizon predictions and integrating trajectory planning.<n>This is because conventional video diffusion models rely on global joint distribution modeling of fixed-length frame sequences.<n>We propose Epona, an autoregressive world model that enables localized distribution modeling.
  arXiv  Detail & Related papers  (2025-06-30T17:56:35Z)
• ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model [52.02220087880269]
  We propose an extension of ManiGaussian framework that improves bimanual manipulation by digesting multi-task scene dynamics through a hierarchical world model.<n>Our method significantly outperforms the current state-of-the-art bimanual manipulation techniques by an improvement of 20.2% in 10 simulated tasks, and 60% success rate on average in 9 challenging real-world tasks.
  arXiv  Detail & Related papers  (2025-06-24T17:59:06Z)
• Consistent World Models via Foresight Diffusion [56.45012929930605]
  We argue that a key bottleneck in learning consistent diffusion-based world models lies in the suboptimal predictive ability.<n>We propose Foresight Diffusion (ForeDiff), a diffusion-based world modeling framework that enhances consistency by decoupling condition understanding from target denoising.
  arXiv  Detail & Related papers  (2025-05-22T10:01:59Z)
• Vid2World: Crafting Video Diffusion Models to Interactive World Models [38.270098691244314]
  Vid2World is a general approach for leveraging and transferring pre-trained video diffusion models into interactive world models.<n>It performs casualization of a pre-trained video diffusion model by crafting its architecture and training objective to enable autoregressive generation.<n>It introduces a causal action guidance mechanism to enhance action controllability in the resulting interactive world model.
  arXiv  Detail & Related papers  (2025-05-20T13:41:45Z)
• Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets [7.667819384855409]
  We present Unified World Models (UWM), a framework that allows for leveraging both video and action data for policy learning.<n>By controlling each diffusion timestep, UWM can flexibly represent a policy, a forward dynamics, an inverse dynamics, and a video generator.<n>Our results suggest that UWM offers a promising step toward harnessing large, heterogeneous datasets for scalable robot learning.
  arXiv  Detail & Related papers  (2025-04-03T17:38:59Z)
• HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model [54.64088247291416]
  A fundamental objective of manipulation policy design is to endow robots to comprehend human instructions, reason about scene cues, and execute generalized actions in dynamic environments.<n>Recent autoregressive vision-language-action (VLA) methods inherit common-sense reasoning capabilities from vision-language models (VLMs) for next action-token prediction.<n>We introduce HybridVLA, a unified framework that absorbs the continuous nature of diffusion-based actions and the contextual reasoning of autoregression.
  arXiv  Detail & Related papers  (2025-03-13T17:59:52Z)
• HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation [54.03004125910057]
  We show that hierarchical vision-language-action models can be more effective in utilizing off-domain data than standard monolithic VLA models.<n>We show that, with the hierarchical design, the high-level VLM can transfer across significant domain gaps between the off-domain finetuning data and real-robot testing scenarios.
  arXiv  Detail & Related papers  (2025-02-08T07:50:22Z)
• VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation [79.00294932026266]
  VidMan is a novel framework that employs a two-stage training mechanism to enhance stability and improve data utilization efficiency. Our framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7% relative improvement, and demonstrates over 9% precision gains on the OXE small-scale dataset.
  arXiv  Detail & Related papers  (2024-11-14T03:13:26Z)
• IRASim: A Fine-Grained World Model for Robot Manipulation [24.591694756757278]
  We present IRASim, a novel world model capable of generating videos with fine-grained robot-object interaction details.<n>We train a diffusion transformer and introduce a novel frame-level action-conditioning module within each transformer block to explicitly model and strengthen the action-frame alignment.
  arXiv  Detail & Related papers  (2024-06-20T17:50:16Z)
• Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion [36.321494200830244]
  Copilot4D is a novel world modeling approach that first tokenizes sensor observations with VQVAE, then predicts the future via discrete diffusion. Our results demonstrate that discrete diffusion on tokenized agent experience can unlock the power of GPT-like unsupervised learning for robotics.
  arXiv  Detail & Related papers  (2023-11-02T06:21:56Z)
• Interactive Character Control with Auto-Regressive Motion Diffusion Models [18.727066177880708]
  We propose A-MDM (Auto-regressive Motion Diffusion Model) for real-time motion synthesis. Our conditional diffusion model takes an initial pose as input, and auto-regressively generates successive motion frames conditioned on previous frame. We introduce a suite of techniques for incorporating interactive controls into A-MDM, such as task-oriented sampling, in-painting, and hierarchical reinforcement learning.
  arXiv  Detail & Related papers  (2023-06-01T07:48:34Z)
• An Efficient Membership Inference Attack for the Diffusion Model by Proximal Initialization [58.88327181933151]
  In this paper, we propose an efficient query-based membership inference attack (MIA) Experimental results indicate that the proposed method can achieve competitive performance with only two queries on both discrete-time and continuous-time diffusion models. To the best of our knowledge, this work is the first to study the robustness of diffusion models to MIA in the text-to-speech task.
  arXiv  Detail & Related papers  (2023-05-26T16:38:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.