Full-Body Visual Self-Modeling of Robot Morphologies

• URL: http://arxiv.org/abs/2111.06389v1
• Date: Thu, 11 Nov 2021 18:58:07 GMT
• Title: Full-Body Visual Self-Modeling of Robot Morphologies
• Authors: Boyuan Chen, Robert Kwiatkowski, Carl Vondrick, Hod Lipson
• Abstract summary: Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. Here, we propose that instead of directly modeling forward-kinematics, a more useful form of self-modeling is one that could answer space occupancy queries.
• Score: 29.76701883250049
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. These "self-models" allow robots to consider outcomes of multiple possible future actions, without trying them out in physical reality. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. However, forward-kinema\-tics models can only predict limited aspects of the morphology, such as the position of end effectors or velocity of joints and masses. A key challenge is to model the entire morphology and kinematics, without prior knowledge of what aspects of the morphology will be relevant to future tasks. Here, we propose that instead of directly modeling forward-kinematics, a more useful form of self-modeling is one that could answer space occupancy queries, conditioned on the robot's state. Such query-driven self models are continuous in the spatial domain, memory efficient, fully differentiable and kinematic aware. In physical experiments, we demonstrate how a visual self-model is accurate to about one percent of the workspace, enabling the robot to perform various motion planning and control tasks. Visual self-modeling can also allow the robot to detect, localize and recover from real-world damage, leading to improved machine resiliency. Our project website is at: https://robot-morphology.cs.columbia.edu/

Related papers

• Differentiable Robot Rendering [45.23538293501457]
  We introduce differentiable robot rendering, a method allowing the visual appearance of a robot body to be directly differentiable with respect to its control parameters. We demonstrate its capability and usage in applications including reconstruction of robot poses from images and controlling robots through vision language models.
  arXiv  Detail & Related papers  (2024-10-17T17:59:02Z)
• Teaching Robots to Build Simulations of Themselves [7.886658271375681]
  We introduce a self-supervised learning framework to enable robots model and predict their morphology, kinematics and motor control using only brief raw video data. By observing their own movements, robots learn an ability to simulate themselves and predict their spatial motion for various tasks.
  arXiv  Detail & Related papers  (2023-11-20T20:03:34Z)
• High-Degrees-of-Freedom Dynamic Neural Fields for Robot Self-Modeling and Motion Planning [6.229216953398305]
  A robot self-model is a representation of the robot's physical morphology that can be used for motion planning tasks. We propose a new encoder-based neural density field architecture for dynamic object-centric scenes conditioned on high numbers of degrees of freedom. In a 7-DOF robot test setup, the learned self-model achieves a Chamfer-L2 distance of 2% of the robot's dimension workspace.
  arXiv  Detail & Related papers  (2023-10-05T16:01:29Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• RT-1: Robotics Transformer for Real-World Control at Scale [98.09428483862165]
  We present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.
  arXiv  Detail & Related papers  (2022-12-13T18:55:15Z)
• On the Origins of Self-Modeling [27.888203008100113]
  Self-Modeling is the process by which an agent, such as an animal or machine, learns to create a predictive model of its own dynamics. Here, we quantify the benefits of such self-modeling against the complexity of the robot.
  arXiv  Detail & Related papers  (2022-09-05T15:27:04Z)
• RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks [32.00371492516123]
  We present a model-based planning framework for modeling and manipulating elasto-plastic objects. Our system, RoboCraft, learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes.
  arXiv  Detail & Related papers  (2022-05-05T20:28:15Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)
• Model-Based Visual Planning with Self-Supervised Functional Distances [104.83979811803466]
  We present a self-supervised method for model-based visual goal reaching. Our approach learns entirely using offline, unlabeled data. We find that this approach substantially outperforms both model-free and model-based prior methods.
  arXiv  Detail & Related papers  (2020-12-30T23:59:09Z)
• Learning Predictive Models From Observation and Interaction [137.77887825854768]
  Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works. However, learning a model that captures the dynamics of complex skills represents a major challenge. We propose a method to augment the training set with observational data of other agents, such as humans.
  arXiv  Detail & Related papers  (2019-12-30T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.