PLUME: Probabilistic Latent Unified World Modeling and Parameter Estimation for Multi-Finger Manipulation

• URL: http://arxiv.org/abs/2606.11396v1
• Date: Tue, 09 Jun 2026 19:36:25 GMT
• Title: PLUME: Probabilistic Latent Unified World Modeling and Parameter Estimation for Multi-Finger Manipulation
• Abstract summary: We propose Probabilistic Latent Unified world Modeling and parameter Estimation (PLUME)<n>PLUME learns to evolve a belief over parameter values as well as the system dynamics conditioned on those parameters.<n>We evaluate our method on simulated screwdriver turning, valve turning, bucket lifting, and disk flicking tasks.
• Score: 15.86252454772027
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dexterous manipulation with multi-finger hands can be sensitive to physical parameters such as object shape, pose, and friction coefficients. While simulation enables large-scale data collection with known parameter values, simulation-trained policies must still handle uncertainty at deployment, where the true parameters and therefore the true dynamics are unknown. Standard domain randomization strategies may be insufficient for precise tasks like screwdriver turning, as manipulation strategies may need to change depending on specific parameter values. To address this, we propose Probabilistic Latent Unified world Modeling and parameter Estimation (PLUME), a world model that jointly learns to evolve a belief over parameter values as well as the system dynamics conditioned on those parameters. We learn a latent space to jointly represent multiple qualitatively different physical parameters along with rewards, themselves functions of partially-observable variables, to inform planning. Our novel learning framework leads to efficient alignment of the world model to true dynamics through online parameter inference as opposed to re-training or fine-tuning. We evaluate our method on simulated screwdriver turning, valve turning, bucket lifting, and disk flicking tasks, as well as a hardware screwdriver turning task, where we achieve successful zero-shot transfer of our simulation-trained policy and outperform state-of-the-art offline reinforcement learning and world-model-augmented behavior cloning baselines. Please see our website at https://plume-world-model.github.io for videos.

Related papers

• Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning [2.071516130824992]
  Transferring reinforcement learning policies from simulation to Real-World robots remains a major challenge.<n>We propose a Real2Sim2Real pipeline that improves Sim2Real transfer for torque-controlled robotic arms.<n>This work highlights the effectiveness of accurate physical modelling in enabling stable and generalizable torque-based reinforcement learning policies.
  arXiv  Detail & Related papers  (2026-08-23T22:25:33Z)
• Dynamics Models for Offline Hyperparameter Selection in Real-World RL [18.05255852818742]
  We present the first application of calibration models in a real-world industrial setting: a municipal water treatment plant.<n>Our results show that these models can generate realistic long-horizon rollouts and recover meaningful hyper parameter sensitivity trends.<n>Overall, our findings provide a proof of concept for using offline dynamics models to support RL deployment in real-world environments.
  arXiv  Detail & Related papers  (2026-08-11T18:54:51Z)
• D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping [66.22412592525369]
  We introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine.<n>We show that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values.<n>Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping.
  arXiv  Detail & Related papers  (2026-03-01T15:32:04Z)
• Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulation [17.786858357661604]
  Phys2Real is a real-to-sim-to-real RL pipeline that combines vision-language model (VLM)-inferred physical parameter estimates with interactive adaptation through uncertainty-aware fusion.<n>Our approach consists of three core components: (1) high-fidelity geometric reconstruction with 3D Gaussian splatting, (2) VLM-inferred prior over physical parameters, and (3) online physical parameter estimation from interaction data.
  arXiv  Detail & Related papers  (2025-10-13T17:51:23Z)
• PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policies [7.376615925443845]
  We introduce PrivilegedDreamer, a model-based reinforcement learning framework that extends the existing model-based approach by incorporating an explicit parameter estimation module.<n>Our empirical analysis on five diverse HIP-MDP tasks demonstrates that PrivilegedDreamer outperforms state-of-the-art model-based, model-free, and do- main adaptation learning algorithms.
  arXiv  Detail & Related papers  (2025-02-17T02:46:02Z)
• Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identifications [23.94013806312391]
  We propose a novel approach that dynamically adjusts simulation environment parameters online using in-context learning.<n>We validate our approach across two tasks: object scooping and table air hockey.<n>Our approach delivers efficient and smooth system identification, advancing the deployment of robots in dynamic real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-27T07:13:38Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• Learning minimal representations of stochastic processes with variational autoencoders [52.99137594502433]
  We introduce an unsupervised machine learning approach to determine the minimal set of parameters required to describe a process. Our approach enables for the autonomous discovery of unknown parameters describing processes.
  arXiv  Detail & Related papers  (2023-07-21T14:25:06Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)
• Dream to Explore: Adaptive Simulations for Autonomous Systems [3.0664963196464448]
  We tackle the problem of learning to control dynamical systems by applying Bayesian nonparametric methods. By employing Gaussian processes to discover latent world dynamics, we mitigate common data efficiency issues observed in reinforcement learning. Our algorithm jointly learns a world model and policy by optimizing a variational lower bound of a log-likelihood.
  arXiv  Detail & Related papers  (2021-10-27T04:27:28Z)
• Auto-Tuned Sim-to-Real Transfer [143.44593793640814]
  Policies trained in simulation often fail when transferred to the real world. Current approaches to tackle this problem, such as domain randomization, require prior knowledge and engineering. We propose a method for automatically tuning simulator system parameters to match the real world.
  arXiv  Detail & Related papers  (2021-04-15T17:59:55Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.