Visual Sculpting: Visually-Aligned Planning Representations for Long-Horizon Robot Clay Sculpting

• URL: http://arxiv.org/abs/2605.17556v1
• Date: Sun, 17 May 2026 17:37:34 GMT
• Title: Visual Sculpting: Visually-Aligned Planning Representations for Long-Horizon Robot Clay Sculpting
• Abstract summary: We present a method for modeling the dynamics of deformable materials and planning for robotic sculpting in a representation that is visually-aligned.<n>Our actions are represented as parametrized pushes into clay with a single end-effector, which proved to be suitable for long-horizon (>100 actions) clay relief sculptures.
• Score: 12.937086930403671
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Clay sculpting is a nuanced, artistic task involving dexterous manipulation with long-horizon planning to achieve high-level goals. As a robotics problem, we formulate clay sculpting as a shape-to-shape matching challenge. Prior deformable object manipulation work either requires retraining a policy per goal or relies on dynamics models which represent state as sparse point clouds which do not capture important clay features, such as textures, well. We present a method for modeling the dynamics of deformable materials and planning for robotic sculpting in a representation that is visually-aligned, capturing lighting and texture features. With three different deformable materials and various end-effectors, we demonstrate that our dynamics model is comparable in performance to the state-of-the-art with the added benefit of being compatible with visual planning. Our actions are represented as parametrized pushes into clay with a single end-effector, which proved to be suitable for long-horizon (>100 actions) clay relief sculptures. Lastly, we show the benefits of planning in a visually-aligned representation, but also provide analysis providing evidence as to why this representation is challenging to plan in compared to 3D representations.

Related papers

• PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoning [24.361346589238035]
  We present PhysGraph, a framework that unifies symbolic reasoning with structured 3D geometry to model kinematic and physical properties in cluttered scenes.<n> PhysGraph achieves state-of-the-art results in semantic segmentation, multi-object mass estimation, and articulation prediction.
  arXiv  Detail & Related papers  (2026-06-07T14:50:44Z)
• SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy [6.937243101289336]
  SculptDiff is a goal-conditioned diffusion-based imitation learning framework. It works with point cloud state observations to learn clay sculpting policies for a variety of target shapes. To the best of our knowledge this is the first real-world method that successfully learns manipulation policies for 3D deformable objects.
  arXiv  Detail & Related papers  (2024-03-15T15:34:59Z)
• TriHuman : A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis [76.73338151115253]
  TriHuman is a novel human-tailored, deformable, and efficient tri-plane representation. We non-rigidly warp global ray samples into our undeformed tri-plane texture space. We show how such a tri-plane feature representation can be conditioned on the skeletal motion to account for dynamic appearance and geometry changes.
  arXiv  Detail & Related papers  (2023-12-08T16:40:38Z)
• SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation [8.517406772939292]
  State representation for materials that exhibit plastic behavior, like modeling clay or bread dough, is difficult because they permanently deform under stress and are constantly changing shape. We propose a system that uses point clouds as the state representation and leverages pre-trained point cloud reconstruction Transformer to learn a latent dynamics model to predict material deformations given a grasp action.
  arXiv  Detail & Related papers  (2023-09-15T19:27:44Z)
• Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces [6.098254376499899]
  We boost the realism of our synthetic faces by introducing dynamic skin wrinkles in response to facial expressions. Our key contribution is an approach that produces realistic wrinkles across a large and diverse population of digital humans. We also introduce the 300W-winks evaluation subset and the Pexels dataset of closed eyes and winks.
  arXiv  Detail & Related papers  (2022-10-05T18:00:13Z)
• Drivable Volumetric Avatars using Texel-Aligned Features [52.89305658071045]
  Photo telepresence requires both high-fidelity body modeling and faithful driving to enable dynamically synthesized appearance. We propose an end-to-end framework that addresses two core challenges in modeling and driving full-body avatars of real people.
  arXiv  Detail & Related papers  (2022-07-20T09:28:16Z)
• 3D Neural Scene Representations for Visuomotor Control [78.79583457239836]
  We learn models for dynamic 3D scenes purely from 2D visual observations. A dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks.
  arXiv  Detail & Related papers  (2021-07-08T17:49:37Z)
• Real-time Deep Dynamic Characters [95.5592405831368]
  We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing. We show that our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches.
  arXiv  Detail & Related papers  (2021-05-04T23:28:55Z)
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors [69.02332607843569]
  PriSMONet is a novel approach for learning Multi-Object 3D scene decomposition and representations from single images. A recurrent encoder regresses a latent representation of 3D shape, pose and texture of each object from an input RGB image. We evaluate the accuracy of our model in inferring 3D scene layout, demonstrate its generative capabilities, assess its generalization to real images, and point out benefits of the learned representation.
  arXiv  Detail & Related papers  (2020-10-08T14:49:23Z)
• Learning to Caricature via Semantic Shape Transform [95.25116681761142]
  We propose an algorithm based on a semantic shape transform to produce shape exaggerations. We show that the proposed framework is able to render visually pleasing shape exaggerations while maintaining their facial structures.
  arXiv  Detail & Related papers  (2020-08-12T03:41:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.