Learning to Predict Contact Force Distributions from Vision Leveraging Object Geometry Priors

• URL: http://arxiv.org/abs/2608.00464v1
• Date: Sat, 01 Aug 2026 06:11:50 GMT
• Title: Learning to Predict Contact Force Distributions from Vision Leveraging Object Geometry Priors
• Abstract summary: We use a rigid-body simulator commonly adopted in robotics to collect paired data of vision and forces.<n>Unlike simulators that output noisy point forces, humans are able to make consistent predictions even in unfamiliar situations.<n>We construct a model that predicts three-dimensional force distributions from a single RGB image of piled daily objects.
• Score: 3.683654272386691
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Based on vision and prior experience, humans can make rough physical predictions and adjust their manipulation strategies. This paper aims to endow robots with a similar ability. To collect paired data of vision and forces, we use a rigid-body simulator commonly adopted in robotics. However, unlike simulators that output noisy point forces, humans are able to make consistent predictions even in unfamiliar situations. Based on this observation, we hypothesize that predicting smooth force distributions rather than raw point forces can improve both force prediction itself and downstream task performance. To validate this hypothesis, we construct a model that predicts three-dimensional force distributions from a single RGB image of piled daily objects. The target distribution is generated by applying statistical smoothing to point forces obtained from the simulator. Moreover, by incorporating object geometry into the smoothing process, we aim to account for variations in contact states and achieve more consistent vision-based predictions. We conduct extensive evaluations in both simulation and real environments. Results show that our approach improves prediction accuracy, enhances downstream task performance through smoothing, and further benefits from geometry-guided smoothing. Remarkably, the trained model generalizes effectively to real-world scenes despite being trained solely in simulation.

Related papers

• Learning Physics-Guided Residual Dynamics for Deformable Object Simulation [19.104444935269616]
  Physics-Guided Residual Dynamics (PGRD) is a hybrid simulation framework that combines the advantages of physics-based and learning-based approaches.<n>We show that PGRD produces more accurate results than both purely physics-based and learning-based methods on a set of diverse real-world deformable objects.
  arXiv  Detail & Related papers  (2026-07-15T05:15:43Z)
• Bridging the sim2real gap in the table tennis robot with a transformer-based ball states predictor [42.252953556229066]
  Robotic table tennis is a representative benchmark for high-speed, closed-loop robotic control in dynamic environments.<n>We propose a transformer-based framework for table tennis ball state prediction.<n>We collect a large-scale real-world dataset from players of varying skill levels and diverse ball cannon configurations.
  arXiv  Detail & Related papers  (2026-06-09T21:35:59Z)
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model [76.08429266631823]
  We propose an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM)<n>URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction.<n> Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-11-02T13:45:51Z)
• Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers [3.951575888190684]
  This study investigates a transformer adaptation for video prediction with a simple end-to-end approach, comparing various selftemporal-attention layouts.<n>We introduce a simple yet effective transformer for autoregressive video prediction, utilizing continuous pixel-space representations for video prediction horizon.
  arXiv  Detail & Related papers  (2025-10-23T17:58:45Z)
• Learning Contact Dynamics for Control with Action-conditioned Face Interaction Graph Networks [4.942750934906048]
  We present a learnable physics simulator that provides accurate motion and force-torque prediction of robot end effectors in contact-rich manipulation.<n>The proposed model extends the state-of-the-art GNN-based simulator with novel node and edge types.
  arXiv  Detail & Related papers  (2025-09-15T17:15:31Z)
• Unified Human Localization and Trajectory Prediction with Monocular Vision [64.19384064365431]
  MonoTransmotion is a Transformer-based framework that uses only a monocular camera to jointly solve localization and prediction tasks.<n>We show that by jointly training both tasks with our unified framework, our method is more robust in real-world scenarios made of noisy inputs.
  arXiv  Detail & Related papers  (2025-03-05T14:18:39Z)
• What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [57.92924256181857]
  We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
  arXiv  Detail & Related papers  (2023-11-03T18:09:08Z)
• DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation [81.11585774044848]
  We present DeepSimHO, a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization.
  arXiv  Detail & Related papers  (2023-10-11T05:34:36Z)
• A Control-Centric Benchmark for Video Prediction [69.22614362800692]
  We propose a benchmark for action-conditioned video prediction in the form of a control benchmark. Our benchmark includes simulated environments with 11 task categories and 310 task instance definitions. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling.
  arXiv  Detail & Related papers  (2023-04-26T17:59:45Z)
• Object and Relation Centric Representations for Push Effect Prediction [18.990827725752496]
  Pushing is an essential non-prehensile manipulation skill used for tasks ranging from pre-grasp manipulation to scene rearrangement. We propose a graph neural network based framework for effect prediction and parameter estimation of pushing actions. Our framework is validated both in real and simulated environments containing different shaped multi-part objects connected via different types of joints and objects with different masses.
  arXiv  Detail & Related papers  (2021-02-03T15:09:12Z)
• Predicting the Physical Dynamics of Unseen 3D Objects [65.49291702488436]
  We focus on predicting the dynamics of 3D objects on a plane that have just been subjected to an impulsive force. Our approach can generalize to object shapes and initial conditions that were unseen during training. Our model can support training with data from both a physics engine or the real world.
  arXiv  Detail & Related papers  (2020-01-16T06:27:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.