How Physics and Background Attributes Impact Video Transformers in Robotic Manipulation: A Case Study on Planar Pushing

• URL: http://arxiv.org/abs/2310.02044v4
• Date: Wed, 28 Aug 2024 09:34:33 GMT
• Title: How Physics and Background Attributes Impact Video Transformers in Robotic Manipulation: A Case Study on Planar Pushing
• Authors: Shutong Jin, Ruiyu Wang, Muhammad Zahid, Florian T. Pokorny,
• Abstract summary: We investigate how physics attributes and scene background characteristics influence the performance of Video Transformers. We present CloudGripper-Push-1K, a large real-world vision-based robot pushing dataset. We also propose Video Occlusion Transformer (VOT), a generic modular video-transformer-based trajectory prediction framework.
• Score: 8.435401907462245
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As model and dataset sizes continue to scale in robot learning, the need to understand how the composition and properties of a dataset affect model performance becomes increasingly urgent to ensure cost-effective data collection and model performance. In this work, we empirically investigate how physics attributes (color, friction coefficient, shape) and scene background characteristics, such as the complexity and dynamics of interactions with background objects, influence the performance of Video Transformers in predicting planar pushing trajectories. We investigate three primary questions: How do physics attributes and background scene characteristics influence model performance? What kind of changes in attributes are most detrimental to model generalization? What proportion of fine-tuning data is required to adapt models to novel scenarios? To facilitate this research, we present CloudGripper-Push-1K, a large real-world vision-based robot pushing dataset comprising 1278 hours and 460,000 videos of planar pushing interactions with objects with different physics and background attributes. We also propose Video Occlusion Transformer (VOT), a generic modular video-transformer-based trajectory prediction framework which features 3 choices of 2D-spatial encoders as the subject of our case study. The dataset and source code are available at https://cloudgripper.org.

Related papers

• AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation [30.367498271886866]
  This paper introduces AdaptiGraph, a learning-based dynamics modeling approach. It enables robots to predict, adapt to, and control a wide array of challenging deformable materials. On prediction and manipulation tasks involving a diverse set of real-world deformable objects, our method exhibits superior prediction accuracy and task proficiency.
  arXiv  Detail & Related papers  (2024-07-10T17:57:04Z)
• TK-Planes: Tiered K-Planes with High Dimensional Feature Vectors for Dynamic UAV-based Scenes [58.180556221044235]
  We present a new approach to bridge the domain gap between synthetic and real-world data for unmanned aerial vehicle (UAV)-based perception. Our formulation is designed for dynamic scenes, consisting of small moving objects or human actions. We evaluate its performance on challenging datasets, including Okutama Action and UG2.
  arXiv  Detail & Related papers  (2024-05-04T21:55:33Z)
• Reasoning-Enhanced Object-Centric Learning for Videos [15.554898985821302]
  We develop a Slot-based Time-Space Transformer with Memory buffer (STATM) to enhance the model's perception ability in complex scenes. As a predictive model, the STATM module also performs well in downstream prediction and Visual Question Answering (VQA) tasks.
  arXiv  Detail & Related papers  (2024-03-22T14:41:55Z)
• Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos [8.012771454339353]
  We propose a novel approach for real-to-sim which tracks rigid objects in 3D from RGB-D images and infers physical properties of the objects. We demonstrate and evaluate our approach on a real-world dataset.
  arXiv  Detail & Related papers  (2023-09-27T14:46:01Z)
• VDT: General-purpose Video Diffusion Transformers via Mask Modeling [62.71878864360634]
  Video Diffusion Transformer (VDT) pioneers the use of transformers in diffusion-based video generation. We propose a unified spatial-temporal mask modeling mechanism, seamlessly integrated with the model, to cater to diverse video generation scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:59:45Z)
• T3VIP: Transformation-based 3D Video Prediction [49.178585201673364]
  We propose a 3D video prediction (T3VIP) approach that explicitly models the 3D motion by decomposing a scene into its object parts. Our model is fully unsupervised, captures the nature of the real world, and the observational cues in image and point cloud domains constitute its learning signals. To the best of our knowledge, our model is the first generative model that provides an RGB-D video prediction of the future for a static camera.
  arXiv  Detail & Related papers  (2022-09-19T15:01:09Z)
• Patch-based Object-centric Transformers for Efficient Video Generation [71.55412580325743]
  We present Patch-based Object-centric Video Transformer (POVT), a novel region-based video generation architecture. We build upon prior work in video prediction via an autoregressive transformer over the discrete latent space of compressed videos. Due to better compressibility of object-centric representations, we can improve training efficiency by allowing the model to only access object information for longer horizon temporal information.
  arXiv  Detail & Related papers  (2022-06-08T16:29:59Z)
• Learning Multi-Object Dynamics with Compositional Neural Radiance Fields [63.424469458529906]
  We present a method to learn compositional predictive models from image observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and graph neural networks. NeRFs have become a popular choice for representing scenes due to their strong 3D prior. For planning, we utilize RRTs in the learned latent space, where we can exploit our model and the implicit object encoder to make sampling the latent space informative and more efficient.
  arXiv  Detail & Related papers  (2022-02-24T01:31:29Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)
• Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction [24.72947291987545]
  Key challenge for an agent learning to interact with the world is to reason about physical properties of objects. We propose a novel approach for modeling the dynamics of a robot's interactions directly from unlabeled 3D point clouds and images.
  arXiv  Detail & Related papers  (2020-08-02T11:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.