DexSIM: Real-time Dexterous Simulation with Unified Causal Video Diffusion

• URL: http://arxiv.org/abs/2605.24630v1
• Date: Sat, 23 May 2026 15:39:38 GMT
• Title: DexSIM: Real-time Dexterous Simulation with Unified Causal Video Diffusion
• Abstract summary: We propose DexSIM, a dexterous simulation framework for simulating dexterous manipulation in real-time.<n>Existing methods lack real-time interactivity and long-term spatial consistency and memory.<n>It also allows new applications such as hand motion transfer and runs at 15.24 FPS real-time interactivity.
• Score: 0.18613536568358358
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress of video diffusion models have enabled extensive simulation of the physical world. While simulation with hand object interaction has been less explored. We propose DexSIM, a dexterous simulation framework for simulating dexterous manipulation in real-time. While previous works utilizing video diffusion and 3D reconstruction focus on navigation, dexterous manipulation has been limited while it has extensive applications for creating interactive experiences with the simulated world and for generating synthetic data for robotics. Existing methods lack real-time interactivity and long-term spatial consistency and memory. We propose a 2-stage training framework for DexSIM. First we train a bi-directional video diffusion model by jointly embedding the hand action trajectory and video in a unified feature space. We utilize gaussian heatmap hand encoding for more accurate hand representation. Then we conduct a roll-out based autoregressive training with updated spatial cache as attention sink for spatial memory, which improves long-term consistency and 3D aware dexterous manipulation simulation. DexSIM outperforms the baseline on pixel and semantic similarity, motion fidelity, and hand projection accuracy. It also allows new applications such as hand motion transfer and runs at 15.24 FPS real-time interactivity.

Related papers

• OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation [51.37436817948304]
  OASIS is a simulation-data-driven framework for humanoid loco-manipulation.<n>It reconstructs realistic object assets from real-world images using a 3D generative model.<n>Under zero-shot deployment, it achieves higher success rates on most tasks than that trained on real-robot teleoperation data.
  arXiv  Detail & Related papers  (2026-06-07T10:01:53Z)
• TIDES: Time-Derivative Event Simulation via Deformable Reconstruction [10.871120244987805]
  We present TIDES, a continuous-time event simulator built on dynamic splatting.<n> TIDES operates on an explicit 3D representation with learnt geometry and motion.<n>It can derive per-pixel intensity dynamics directly from the scene, rather than by differencing rendered frames.
  arXiv  Detail & Related papers  (2026-06-01T10:49:13Z)
• A Synthetic Eye Movement Dataset for Script Reading Detection: Real Trajectory Replay on a 3D Simulator [12.307366979757065]
  We present a pipeline for generating synthetic labeled eye movement video by extracting real human iris trajectories from reference videos.<n>We release 144 sessions (72 reading, 72 conversation) totaling 12 hours of synthetic eye movement video at 25 fps.<n>A matched frame-by-frame comparison reveals that the 3D simulator exhibits bounded sensitivity at reading-scale movements.
  arXiv  Detail & Related papers  (2026-04-07T06:15:48Z)
• DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer [62.18680935878919]
  We introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings into temporally consistent outputs.<n>At its core is a single-step temporally-conditioned enhancer capable of running in online simulators on a single GPU.
  arXiv  Detail & Related papers  (2026-02-27T15:35:30Z)
• SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation [64.3409486422946]
  We present SpriteHand, an autoregressive video generation framework for real-time synthesis of hand-object interaction videos.<n>Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence.<n> Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.
  arXiv  Detail & Related papers  (2025-12-01T18:13:40Z)
• Hybrid Neural-MPM for Interactive Fluid Simulations in Real-Time [57.30651532625017]
  We present a novel hybrid method that integrates numerical simulation, neural physics, and generative control.<n>Our system demonstrates robust performance across diverse 2D/3D scenarios, material types, and obstacle interactions.<n>We promise to release both models and data upon acceptance.
  arXiv  Detail & Related papers  (2025-05-25T01:27:18Z)
• X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real [20.561250366126625]
  X-Sim is a real-to-sim-to-real framework that uses object motion as a dense and transferable signal for learning robot policies.<n>X-Sim starts by reconstructing a simulation from an RGBD human video and tracking object trajectories to define object-centric rewards.<n>The learned policy is then distilled into an image-conditioned diffusion policy using synthetic rollouts rendered with varied viewpoints and lighting.
  arXiv  Detail & Related papers  (2025-05-11T19:04:00Z)
• Learning 3D-Gaussian Simulators from RGB Videos [20.250137125726265]
  3DGSim is a learned 3D simulator that learns physical interactions from multi-view RGB videos.<n>It unifies 3D scene reconstruction, particle dynamics prediction and video synthesis into an end-to-end trained framework.
  arXiv  Detail & Related papers  (2025-03-31T12:33:59Z)
• Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation [62.5805866419814]
  Vid2Sim is a novel framework that bridges the sim2real gap through a scalable and cost-efficient real2sim pipeline for neural 3D scene reconstruction and simulation.<n>Experiments demonstrate that Vid2Sim significantly improves the performance of urban navigation in the digital twins and real world by 31.2% and 68.3% in success rate.
  arXiv  Detail & Related papers  (2025-01-12T03:01:15Z)
• Inferring Articulated Rigid Body Dynamics from RGBD Video [18.154013621342266]
  We introduce a pipeline that combines inverse rendering with differentiable simulation to create digital twins of real-world articulated mechanisms. Our approach accurately reconstructs the kinematic tree of an articulated mechanism being manipulated by a robot.
  arXiv  Detail & Related papers  (2022-03-20T08:19:02Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.