An Exploration of Embodied Visual Exploration

• URL: http://arxiv.org/abs/2001.02192v2
• Date: Fri, 21 Aug 2020 02:58:53 GMT
• Title: An Exploration of Embodied Visual Exploration
• Authors: Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman
• Abstract summary: Embodied computer vision considers perception for robots in novel, unstructured environments. We present a taxonomy for existing visual exploration algorithms and create a standard framework for benchmarking them. We then perform a thorough empirical study of the four state-of-the-art paradigms using the proposed framework.
• Score: 97.21890864063872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied computer vision considers perception for robots in novel, unstructured environments. Of particular importance is the embodied visual exploration problem: how might a robot equipped with a camera scope out a new environment? Despite the progress thus far, many basic questions pertinent to this problem remain unanswered: (i) What does it mean for an agent to explore its environment well? (ii) Which methods work well, and under which assumptions and environmental settings? (iii) Where do current approaches fall short, and where might future work seek to improve? Seeking answers to these questions, we first present a taxonomy for existing visual exploration algorithms and create a standard framework for benchmarking them. We then perform a thorough empirical study of the four state-of-the-art paradigms using the proposed framework with two photorealistic simulated 3D environments, a state-of-the-art exploration architecture, and diverse evaluation metrics. Our experimental results offer insights and suggest new performance metrics and baselines for future work in visual exploration. Code, models and data are publicly available: https://github.com/facebookresearch/exploring_exploration

Related papers

• Explore until Confident: Efficient Exploration for Embodied Question Answering [32.27111287314288]
  We leverage the strong semantic reasoning capabilities of large vision-language models to efficiently explore and answer questions. We propose a method that first builds a semantic map of the scene based on depth information and via visual prompting of a VLM. Next, we use conformal prediction to calibrate the VLM's question answering confidence, allowing the robot to know when to stop exploration.
  arXiv  Detail & Related papers  (2024-03-23T22:04:03Z)
• Embodied Agents for Efficient Exploration and Smart Scene Description [47.82947878753809]
  We tackle a setting for visual navigation in which an autonomous agent needs to explore and map an unseen indoor environment. We propose and evaluate an approach that combines recent advances in visual robotic exploration and image captioning on images. Our approach can generate smart scene descriptions that maximize semantic knowledge of the environment and avoid repetitions.
  arXiv  Detail & Related papers  (2023-01-17T19:28:01Z)
• Robot Active Neural Sensing and Planning in Unknown Cluttered Environments [0.0]
  Active sensing and planning in unknown, cluttered environments is an open challenge for robots intending to provide home service, search and rescue, narrow-passage inspection, and medical assistance. We present the active neural sensing approach that generates the kinematically feasible viewpoint sequences for the robot manipulator with an in-hand camera to gather the minimum number of observations needed to reconstruct the underlying environment. Our framework actively collects the visual RGBD observations, aggregates them into scene representation, and performs object shape inference to avoid unnecessary robot interactions with the environment.
  arXiv  Detail & Related papers  (2022-08-23T16:56:54Z)
• OPEn: An Open-ended Physics Environment for Learning Without a Task [132.6062618135179]
  We will study if models of the world learned in an open-ended physics environment, without any specific tasks, can be reused for downstream physics reasoning tasks. We build a benchmark Open-ended Physics ENvironment (OPEn) and also design several tasks to test learning representations in this environment explicitly. We find that an agent using unsupervised contrastive learning for representation learning, and impact-driven learning for exploration, achieved the best results.
  arXiv  Detail & Related papers  (2021-10-13T17:48:23Z)
• Rapid Exploration for Open-World Navigation with Latent Goal Models [78.45339342966196]
  We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration.
  arXiv  Detail & Related papers  (2021-04-12T23:14:41Z)
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration [102.74649829684617]
  The ability to navigate like a human towards a language-guided target from anywhere in a 3D embodied environment is one of the 'holy grail' goals of intelligent robots. Most visual navigation benchmarks focus on navigating toward a target from a fixed starting point, guided by an elaborate set of instructions that depicts step-by-step. This approach deviates from real-world problems in which human-only describes what the object and its surrounding look like and asks the robot to start navigation from anywhere.
  arXiv  Detail & Related papers  (2021-03-31T15:01:04Z)
• ViNG: Learning Open-World Navigation with Visual Goals [82.84193221280216]
  We propose a learning-based navigation system for reaching visually indicated goals. We show that our system, which we call ViNG, outperforms previously-proposed methods for goal-conditioned reinforcement learning. We demonstrate ViNG on a number of real-world applications, such as last-mile delivery and warehouse inspection.
  arXiv  Detail & Related papers  (2020-12-17T18:22:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.