Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models

• URL: http://arxiv.org/abs/2607.05390v1
• Date: Mon, 06 Jul 2026 17:59:18 GMT
• Title: Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models
• Abstract summary: Deform360 is a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactiles.<n>We evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models.
• Score: 36.26785019314161
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz

Related papers

• 4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting [72.02343855552051]
  Current world action models (WAMs) typically operate on 2D visual data.<n>We leverage an explicit 4D Gaussian Splatting (4DGS) representation that separately models dynamic objects and the static background of a scene.<n>4DGS-WAM lifts 2D observations into a persistent 4D representation so that previously observed static content can be reused during future prediction.
  arXiv  Detail & Related papers  (2026-08-26T16:16:57Z)
• Future Dynamic 3D Reconstruction: A 3D World Model with Disentangled Ego-Motion [69.61280478695205]
  We propose FR3D, a world model that predicts a persistent 3D latent representation for future dynamic 3D reconstruction.<n>Experiments demonstrate FR3D's strong performance for future dynamic 3D reconstruction from monocular observations across multiple datasets, even 2 seconds into the future.
  arXiv  Detail & Related papers  (2026-06-16T17:59:46Z)
• Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects [4.870609209341298]
  We develop a novel model that learns action-conditional dynamics of rigid objects in 3D.<n>We evaluate our model in model-predictive control for non-prehensile manipulation in simulation.
  arXiv  Detail & Related papers  (2026-06-01T09:12:07Z)
• MotionDreamer: Universal Skeletal Motion Generation for 3D Rigged Shapes [52.45449797329548]
  MotionDreamer is a diffusion-based framework for category-agnostic skeletal animation generation from 2D video guidance.<n>We propose a structural-semantic injection mechanism to bridge the gap between 2D visual motion cues and heterogeneous 3D skeletal structures.<n>Our approach significantly outperforms existing methods, setting a new state-of-the-art benchmark for robust and efficient 4D asset generation.
  arXiv  Detail & Related papers  (2026-06-01T00:42:31Z)
• NeuROK: Generative 4D Neural Object Kinematics [35.76495786203886]
  We learn a data-driven kinematic state parameterization for object-centric physical systems.<n>We learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset.
  arXiv  Detail & Related papers  (2026-05-28T17:59:53Z)
• TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects [48.763448779200324]
  This paper presents an affordable framework for collecting 3D datasets of deformable objects using only RGB-D cameras.<n>The proposed method identifies 3D keypoints and robustly tracks their trajectories, incorporating motion consistency constraints.<n>Using this framework, this paper presents a high-quality, large-scale dataset consisting of 6 deformable objects, totaling 110 minutes of trajectory data.
  arXiv  Detail & Related papers  (2026-03-17T18:53:04Z)
• DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling [67.95038177144554]
  We introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video.<n>We employ vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic captions.<n> DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos.
  arXiv  Detail & Related papers  (2025-12-02T18:24:27Z)
• Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos [28.473117601626956]
  We develop a neural dynamics framework that combines object particles and spatial grids in a hybrid representation.<n>We demonstrate that our model learns the dynamics of diverse objects from sparse-view RGB-D recordings of robot-object interactions.<n>Our approach outperforms state-of-the-art learning-based and physics-based simulators, particularly in scenarios with limited camera views.
  arXiv  Detail & Related papers  (2025-06-18T17:59:38Z)
• Learning 3D Persistent Embodied World Models [84.40585374179037]
  We introduce a new persistent embodied world model with an explicit memory of previously generated content.<n>During generation time, our video diffusion model predicts RGB-D video of the future observations of the agent.<n>This generation is then aggregated into a persistent 3D map of the environment.
  arXiv  Detail & Related papers  (2025-05-05T17:59:17Z)
• PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model [23.768571323272152]
  PartRM is a novel 4D reconstruction framework that simultaneously models appearance, geometry, and part-level motion from multi-view images of a static object.<n>We introduce the PartDrag-4D dataset, providing multi-view observations of part-level dynamics across over 20,000 states.<n> Experimental results show that PartRM establishes a new state-of-the-art in part-level motion learning and can be applied in manipulation tasks in robotics.
  arXiv  Detail & Related papers  (2025-03-25T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.