DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation

• URL: http://arxiv.org/abs/2503.16806v1
• Date: Fri, 21 Mar 2025 02:29:52 GMT
• Title: DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation
• Authors: Jiangran Lyu, Ziming Li, Xuesong Shi, Chaoyi Xu, Yizhou Wang, He Wang,
• Abstract summary: We propose a novel framework that enhances action learning by jointly predicting future states and adapting to dynamics variations based on historical trajectories.<n>DyWA achieves an average success rate of 68% in real-world experiments.
• Score: 16.863534382288705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Nonprehensile manipulation is crucial for handling objects that are too thin, large, or otherwise ungraspable in unstructured environments. While conventional planning-based approaches struggle with complex contact modeling, learning-based methods have recently emerged as a promising alternative. However, existing learning-based approaches face two major limitations: they heavily rely on multi-view cameras and precise pose tracking, and they fail to generalize across varying physical conditions, such as changes in object mass and table friction. To address these challenges, we propose the Dynamics-Adaptive World Action Model (DyWA), a novel framework that enhances action learning by jointly predicting future states while adapting to dynamics variations based on historical trajectories. By unifying the modeling of geometry, state, physics, and robot actions, DyWA enables more robust policy learning under partial observability. Compared to baselines, our method improves the success rate by 31.5% using only single-view point cloud observations in the simulation. Furthermore, DyWA achieves an average success rate of 68% in real-world experiments, demonstrating its ability to generalize across diverse object geometries, adapt to varying table friction, and robustness in challenging scenarios such as half-filled water bottles and slippery surfaces.

Related papers

• AdaWorld: Learning Adaptable World Models with Latent Actions [76.50869178593733]
  We propose AdaWorld, an innovative world model learning approach that enables efficient adaptation. Key idea is to incorporate action information during the pretraining of world models. We then develop an autoregressive world model that conditions on these latent actions.
  arXiv  Detail & Related papers  (2025-03-24T17:58:15Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)
• Simplifying Latent Dynamics with Softly State-Invariant World Models [10.722955763425228]
  We introduce the Parsimonious Latent Space Model (PLSM), a world model that regularizes the latent dynamics to make the effect of the agent's actions more predictable. We find that our regularization improves accuracy, generalization, and performance in downstream tasks.
  arXiv  Detail & Related papers  (2024-01-31T13:52:11Z)
• Sequential Neural Barriers for Scalable Dynamic Obstacle Avoidance [7.375976854181687]
  We propose a novel method for compositional learning of Sequential Neural Control Barrier models (SNCBFs) Our approach exploits an important observation: the spatial interaction patterns of multiple dynamic obstacles can be decomposed and predicted through temporal sequences of states for each obstacle. We demonstrate the benefits of the proposed methods in improving dynamic collision avoidance in comparison with existing methods.
  arXiv  Detail & Related papers  (2023-07-06T14:24:17Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models.<n>Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning.<n>Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Learning Robust Dynamics through Variational Sparse Gating [18.476155786474358]
  In environments with many objects, often only a small number of them are moving or interacting at the same time. In this paper, we investigate integrating this inductive bias of sparse interactions into the latent dynamics of world models trained from pixels.
  arXiv  Detail & Related papers  (2022-10-21T02:56:51Z)
• ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation [135.10594078615952]
  We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects. A benchmark contains over 17,000 action trajectories with six types of plush toys and 78 variants. Our model achieves the best performance in geometry, correspondence, and dynamics predictions.
  arXiv  Detail & Related papers  (2022-03-14T04:56:55Z)
• Dream to Explore: Adaptive Simulations for Autonomous Systems [3.0664963196464448]
  We tackle the problem of learning to control dynamical systems by applying Bayesian nonparametric methods. By employing Gaussian processes to discover latent world dynamics, we mitigate common data efficiency issues observed in reinforcement learning. Our algorithm jointly learns a world model and policy by optimizing a variational lower bound of a log-likelihood.
  arXiv  Detail & Related papers  (2021-10-27T04:27:28Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.