Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction

• URL: http://arxiv.org/abs/2205.02834v1
• Date: Thu, 5 May 2022 17:59:36 GMT
• Title: Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction
• Authors: Yining Hong, Kaichun Mo, Li Yi, Leonidas J. Guibas, Antonio Torralba, Joshua B. Tenenbaum, Chuang Gan
• Abstract summary: Given a malfunctional 3D object, humans can perform mental simulations to reason about its functionality and figure out how to fix it. To mimic humans' mental simulation process, we present FixNet, a novel framework that seamlessly incorporates perception and physical dynamics.
• Score: 158.74130075865835
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: This paper studies the problem of fixing malfunctional 3D objects. While previous works focus on building passive perception models to learn the functionality from static 3D objects, we argue that functionality is reckoned with respect to the physical interactions between the object and the user. Given a malfunctional object, humans can perform mental simulations to reason about its functionality and figure out how to fix it. Inspired by this, we propose FixIt, a dataset that contains about 5k poorly-designed 3D physical objects paired with choices to fix them. To mimic humans' mental simulation process, we present FixNet, a novel framework that seamlessly incorporates perception and physical dynamics. Specifically, FixNet consists of a perception module to extract the structured representation from the 3D point cloud, a physical dynamics prediction module to simulate the results of interactions on 3D objects, and a functionality prediction module to evaluate the functionality and choose the correct fix. Experimental results show that our framework outperforms baseline models by a large margin, and can generalize well to objects with similar interaction types.

Related papers

• PhysPart: Physically Plausible Part Completion for Interactable Objects [28.91080122885566]
  We tackle the problem of physically plausible part completion for interactable objects. We propose a diffusion-based part generation model that utilizes geometric conditioning. We also demonstrate our applications in 3D printing, robot manipulation, and sequential part generation.
  arXiv  Detail & Related papers  (2024-08-25T04:56:09Z)
• Physically Compatible 3D Object Modeling from a Single Image [109.98124149566927]
  We present a framework that transforms single images into 3D physical objects. Our framework embeds physical compatibility into the reconstruction process. It consistently enhances the physical realism of 3D models over existing methods.
  arXiv  Detail & Related papers  (2024-05-30T21:59:29Z)
• PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation [62.53760963292465]
  PhysDreamer is a physics-based approach that endows static 3D objects with interactive dynamics. We present our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study.
  arXiv  Detail & Related papers  (2024-04-19T17:41:05Z)
• Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions [8.059133373836913]
  This paper presents an approach that enables a robot to rapidly learn the complete 3D model of a given object for manipulation in unfamiliar orientations. We use an ensemble of partially constructed NeRF models to quantify model uncertainty to determine the next action. Our approach determines when and how to grasp and re-orient an object given its partial NeRF model and re-estimates the object pose to rectify misalignments introduced during the interaction.
  arXiv  Detail & Related papers  (2024-04-02T10:15:06Z)
• DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos [76.01906393673897]
  We propose a self-supervised method to jointly learn 3D motion and depth from monocular videos. Our system contains a depth estimation module to predict depth, and a new decomposed object-wise 3D motion (DO3D) estimation module to predict ego-motion and 3D object motion. Our model delivers superior performance in all evaluated settings.
  arXiv  Detail & Related papers  (2024-03-09T12:22:46Z)
• ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors [73.26004792375556]
  This paper shows that robustness and generalisation to novel scene objects in 3D object-aware character synthesis can be achieved by training a motion model with as few as one reference object. We leverage an implicit feature representation trained on object-only datasets, which encodes an SE(3)-equivariant descriptor field around the object. We demonstrate substantial improvements in 3D virtual character motion and interaction quality and robustness to scenarios with unseen objects.
  arXiv  Detail & Related papers  (2023-08-24T17:59:51Z)
• Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation [25.03715978502528]
  We propose a method for incorporating object interaction and human body dynamics into the task of 3D ego-pose estimation. We use a kinematics model of the human body to represent the entire range of human motion, and a dynamics model of the body to interact with objects inside a physics simulator. This is the first work to estimate a physically valid 3D full-body interaction sequence with objects from egocentric videos.
  arXiv  Detail & Related papers  (2020-11-10T00:06:43Z)
• Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction [24.72947291987545]
  Key challenge for an agent learning to interact with the world is to reason about physical properties of objects. We propose a novel approach for modeling the dynamics of a robot's interactions directly from unlabeled 3D point clouds and images.
  arXiv  Detail & Related papers  (2020-08-02T11:04:49Z)
• Occlusion resistant learning of intuitive physics from videos [52.25308231683798]
  Key ability for artificial systems is to understand physical interactions between objects, and predict future outcomes of a situation. This ability, often referred to as intuitive physics, has recently received attention and several methods were proposed to learn these physical rules from video sequences.
  arXiv  Detail & Related papers  (2020-04-30T19:35:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.