An in-depth experimental study of sensor usage and visual reasoning of robots navigating in real environments

• URL: http://arxiv.org/abs/2111.14666v1
• Date: Mon, 29 Nov 2021 16:27:29 GMT
• Title: An in-depth experimental study of sensor usage and visual reasoning of robots navigating in real environments
• Authors: Assem Sadek, Guillaume Bono, Boris Chidlovskii, Christian Wolf
• Abstract summary: We study the performance and reasoning capacities of real physical agents, trained in simulation and deployed to two different physical environments. We show, that for the PointGoal task, an agent pre-trained on wide variety of tasks and fine-tuned on a simulated version of the target environment can reach competitive performance without modelling any sim2real transfer.
• Score: 20.105395754497202
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual navigation by mobile robots is classically tackled through SLAM plus optimal planning, and more recently through end-to-end training of policies implemented as deep networks. While the former are often limited to waypoint planning, but have proven their efficiency even on real physical environments, the latter solutions are most frequently employed in simulation, but have been shown to be able learn more complex visual reasoning, involving complex semantical regularities. Navigation by real robots in physical environments is still an open problem. End-to-end training approaches have been thoroughly tested in simulation only, with experiments involving real robots being restricted to rare performance evaluations in simplified laboratory conditions. In this work we present an in-depth study of the performance and reasoning capacities of real physical agents, trained in simulation and deployed to two different physical environments. Beyond benchmarking, we provide insights into the generalization capabilities of different agents training in different conditions. We visualize sensor usage and the importance of the different types of signals. We show, that for the PointGoal task, an agent pre-trained on wide variety of tasks and fine-tuned on a simulated version of the target environment can reach competitive performance without modelling any sim2real transfer, i.e. by deploying the trained agent directly from simulation to a real physical robot.

Related papers

• Identifying Terrain Physical Parameters from Vision -- Towards Physical-Parameter-Aware Locomotion and Navigation [33.10872127224328]
  We propose a cross-modal self-supervised learning framework for vision-based environmental physical parameter estimation. We train a physical decoder in simulation to predict friction and stiffness from multi-modal input. The trained network allows the labeling of real-world images with physical parameters in a self-supervised manner to further train a visual network during deployment.
  arXiv  Detail & Related papers  (2024-08-29T14:35:14Z)
• Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks [93.38375271826202]
  We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. We first build a simulator by integrating Gaussian splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.
  arXiv  Detail & Related papers  (2024-06-21T13:48:37Z)
• Learning to navigate efficiently and precisely in real environments [14.52507964172957]
  Embodied AI literature focuses on end-to-end agents trained in simulators like Habitat or AI-Thor. In this work we explore end-to-end training of agents in simulation in settings which minimize the sim2real gap.
  arXiv  Detail & Related papers  (2024-01-25T17:50:05Z)
• Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flight [21.728935597793473]
  This work proposes an adaptive multi-pair contrastive learning strategy for visual representation learning that enables zero-shot scene transfer and real-world deployment. We demonstrate the performance of our approach on the task of agile, vision-based quadrotor flight.
  arXiv  Detail & Related papers  (2023-09-18T15:25:59Z)
• Learning Human-to-Robot Handovers from Point Clouds [63.18127198174958]
  We propose the first framework to learn control policies for vision-based human-to-robot handovers. We show significant performance gains over baselines on a simulation benchmark, sim-to-sim transfer and sim-to-real transfer.
  arXiv  Detail & Related papers  (2023-03-30T17:58:36Z)
• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality [64.51295032956118]
  We train a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups.
  arXiv  Detail & Related papers  (2022-10-25T01:51:36Z)
• Zero-Shot Reinforcement Learning on Graphs for Autonomous Exploration Under Uncertainty [6.42522897323111]
  We present a framework for self-learning a high-performance exploration policy in a single simulation environment. We propose a novel approach that uses graph neural networks in conjunction with deep reinforcement learning.
  arXiv  Detail & Related papers  (2021-05-11T02:42:17Z)
• Reactive Long Horizon Task Execution via Visual Skill and Precondition Models [59.76233967614774]
  We describe an approach for sim-to-real training that can accomplish unseen robotic tasks using models learned in simulation to ground components of a simple task planner. We show an increase in success rate from 91.6% to 98% in simulation and from 10% to 80% success rate in the real-world as compared with naive baselines.
  arXiv  Detail & Related papers  (2020-11-17T15:24:01Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)
• SAPIEN: A SimulAted Part-based Interactive ENvironment [77.4739790629284]
  SAPIEN is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks.
  arXiv  Detail & Related papers  (2020-03-19T00:11:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.