ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation

• URL: http://arxiv.org/abs/2605.27852v2
• Date: Thu, 04 Jun 2026 12:17:18 GMT
• Title: ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation
• Abstract summary: We present ClothTransformer, a framework that reformulates cloth simulation as autoregressive sequence modeling.<n>Existing neural cloth simulators are largely specialized to single scenarios, intrinsically coupled to the mesh discretization.<n>Our approach addresses these limitations through three contributions.
• Score: 28.646171436727233
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified and scalable Transformers have recently achieved remarkable success in modeling diverse phenomena traditionally associated with computer graphics, such as 3D visual effects, rendering processes, and motion in videos. In this work, we take a step further by investigating whether modern Transformer techniques can tackle the challenging task of cloth simulation. To this end, we present ClothTransformer, a framework that reformulates cloth simulation as autoregressive sequence modeling in a learned latent space. Existing neural cloth simulators are largely specialized to single scenarios, intrinsically coupled to the mesh discretization, and lack robust collision handling. Our approach addresses these limitations through three contributions: (1) a unified Transformer architecture that handles diverse scenarios -- body-driven garments, robotic manipulation, and free-fall collisions -- under a single model and achieves approximately $4$--$9{\times}$ lower error than prior state-of-the-art methods across all scenarios; (2) a scalable latent-space formulation that compresses arbitrary-resolution meshes into a fixed-size set of latent tokens, making temporal dynamics computation independent of mesh resolution; and (3) a diverse-scenario high-fidelity penetration-free dataset of ${\sim}$493.4k frames spanning all three settings, which enables a differentiable Continuous Collision Detection (CCD) module to suppress penetration artifacts. Project Page: https://yucrazing.github.io/clothtransformer/

Related papers

• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• ManiSplat: Manipulation Trajectory Synthesis from Monocular Video via Decoupled 3D Gaussian Splatting [28.256286705954846]
  Reconstructing dynamic and interactive 3D scenes from real-world observations remains a fundamental challenge in computer vision and robotics.<n>We introduce ManiSplat, a unified framework that reconstructs controllable and decoupled Gaussian digital twins directly from robotic videos.<n>Our approach reconstructs interaction-driven dynamic scenes with high fidelity and controllability, effectively supporting downstream robotic tasks and policy learning.
  arXiv  Detail & Related papers  (2026-06-09T09:55:58Z)
• Part-Level 3D Gaussian Vehicle Generation with Joint and Hinge Axis Estimation [19.45064081855758]
  Simulation is essential for autonomous driving, yet current frameworks often model vehicles as rigid assets and fail to capture part-level articulation.<n>We propose a generative framework that synthesizes an animatable 3D Gaussian vehicle from a single image or sparse multi-view input.<n>Our method addresses two challenges: (i) large 3D asset generators are optimized for static quality but not articulation, leading to distortions at part boundaries when animated; and (ii) segmentation alone cannot provide the kinematic parameters required for motion.
  arXiv  Detail & Related papers  (2026-04-06T18:16:12Z)
• MotionAnymesh: Physics-Grounded Articulation for Simulation-Ready Digital Twins [25.25391483526521]
  MotionAnymesh is a framework that seamlessly transforms unstructured static meshes into simulation-ready digital twins.<n>Our method features a kinematic-aware part segmentation module that grounds VLM reasoning with explicit SP4D physical priors.
  arXiv  Detail & Related papers  (2026-03-13T12:30:42Z)
• Structural Action Transformer for 3D Dexterous Manipulation [80.07649565189035]
  Cross-embodiment skill transfer is a challenge for high-DoF robotic hands.<n>Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations.<n>This paper proposes a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective.
  arXiv  Detail & Related papers  (2026-03-04T11:38:12Z)
• DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer [62.18680935878919]
  We introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings into temporally consistent outputs.<n>At its core is a single-step temporally-conditioned enhancer capable of running in online simulators on a single GPU.
  arXiv  Detail & Related papers  (2026-02-27T15:35:30Z)
• Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries [51.028432812178266]
  Transolver-3 is a new member of the Transolver family designed for high-fidelity physics simulations.<n>We show that Transolver-3 is capable of handling meshes with over 160 million cells, achieving impressive performance across three challenging simulation benchmarks.
  arXiv  Detail & Related papers  (2026-02-04T16:52:44Z)
• AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation [45.753757870577196]
  We introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning.<n>We show that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior art frequently collapses.
  arXiv  Detail & Related papers  (2026-02-04T15:42:58Z)
• StdGEN++: A Comprehensive System for Semantic-Decomposed 3D Character Generation [57.06461272772509]
  StdGEN++ is a novel and comprehensive system for generating high-fidelity, semantically decomposed 3D characters from diverse inputs.<n>It achieves state-of-the-art performance, significantly outperforming existing methods in geometric accuracy and semantic disentanglement.<n>The resulting structural independence unlocks advanced downstream capabilities, including non-destructive editing, physics-compliant animation, and gaze tracking.
  arXiv  Detail & Related papers  (2026-01-12T15:41:27Z)
• Surrogate compliance modeling enables reinforcement learned locomotion gaits for soft robots [5.193389420813379]
  Adaptive morphogenetic robots adapt their morphology and control policies to meet changing tasks and environmental conditions.<n>Many such systems leverage soft components, which enable shape morphing but also introduce simulation and control challenges.<n>Here, we present a surrogate compliance modeling approach: rather than explicitly modeling soft-body physics, we introduce indirect variables representing soft-material deformation within a rigid-body simulator.
  arXiv  Detail & Related papers  (2025-12-08T02:52:42Z)
• SimVS: Simulating World Inconsistencies for Robust View Synthesis [102.83898965828621]
  We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture.<n>We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations.
  arXiv  Detail & Related papers  (2024-12-10T17:35:12Z)
• MoDA: Modeling Deformable 3D Objects from Casual Videos [84.29654142118018]
  We propose neural dual quaternion blend skinning (NeuDBS) to achieve 3D point deformation without skin-collapsing artifacts. In the endeavor to register 2D pixels across different frames, we establish a correspondence between canonical feature embeddings that encodes 3D points within the canonical space. Our approach can reconstruct 3D models for humans and animals with better qualitative and quantitative performance than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-17T13:49:04Z)
• gradSim: Differentiable simulation for system identification and visuomotor control [66.37288629125996]
  We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering. Our unified graph enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision.
  arXiv  Detail & Related papers  (2021-04-06T16:32:01Z)
• Deep Deformation Detail Synthesis for Thin Shell Models [47.442883859643004]
  In physics-based cloth animation, rich folds and detailed wrinkles are achieved at the cost of expensive computational resources and huge labor tuning. We develop a temporally and spatially as-consistent-as-possible deformation representation (named TS-ACAP) and a DeformTransformer network to learn the mapping from low-resolution meshes to detailed ones. Our method is able to produce reliable and realistic animations in various datasets at high frame rates: 10 35 times faster than physics-based simulation, with superior detail synthesis abilities than existing methods.
  arXiv  Detail & Related papers  (2021-02-23T08:09:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.