AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale

• URL: http://arxiv.org/abs/2404.03482v2
• Date: Thu, 11 Jul 2024 16:00:05 GMT
• Title: AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale
• Authors: Adam Pardyl, Michał Wronka, Maciej Wołczyk, Kamil Adamczewski, Tomasz Trzciński, Bartosz Zieliński,
• Abstract summary: Active Visual Exploration (AVE) is a task that involves dynamically selecting observations (glimpses) Existing mobile platforms equipped with optical zoom capabilities can capture glimpses of arbitrary positions and scales. AdaGlimpse uses Soft Actor-Critic, a reinforcement learning algorithm tailored for exploration tasks, to select glimpses of arbitrary position and scale.
• Score: 2.462953128215088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Active Visual Exploration (AVE) is a task that involves dynamically selecting observations (glimpses), which is critical to facilitate comprehension and navigation within an environment. While modern AVE methods have demonstrated impressive performance, they are constrained to fixed-scale glimpses from rigid grids. In contrast, existing mobile platforms equipped with optical zoom capabilities can capture glimpses of arbitrary positions and scales. To address this gap between software and hardware capabilities, we introduce AdaGlimpse. It uses Soft Actor-Critic, a reinforcement learning algorithm tailored for exploration tasks, to select glimpses of arbitrary position and scale. This approach enables our model to rapidly establish a general awareness of the environment before zooming in for detailed analysis. Experimental results demonstrate that AdaGlimpse surpasses previous methods across various visual tasks while maintaining greater applicability in realistic AVE scenarios.

Related papers

• Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal Sensors [49.99728312519117]
  The aim of this work is to establish how accurately a recent semantic-based active perception model is able to complete visual tasks that are regularly performed by humans. This model exploits the ability of current object detectors to localize and classify a large number of object classes and to update a semantic description of a scene across multiple fixations. In the task of scene exploration, the semantic-based method demonstrates superior performance compared to the traditional saliency-based model.
  arXiv  Detail & Related papers  (2024-04-16T18:15:57Z)
• Aligning Knowledge Graph with Visual Perception for Object-goal Navigation [16.32780793344835]
  We propose the Aligning Knowledge Graph with Visual Perception (AKGVP) method for object-goal navigation. Our approach introduces continuous modeling of the hierarchical scene architecture and leverages visual-language pre-training to align natural language description with visual perception. The integration of a continuous knowledge graph architecture and multimodal feature alignment empowers the navigator with a remarkable zero-shot navigation capability.
  arXiv  Detail & Related papers  (2024-02-29T06:31:18Z)
• Active Sensing with Predictive Coding and Uncertainty Minimization [0.0]
  We present an end-to-end procedure for embodied exploration inspired by two biological computations. We first demonstrate our approach in a maze navigation task and show that it can discover the underlying transition distributions and spatial features of the environment. We show that our model builds unsupervised representations through exploration that allow it to efficiently categorize visual scenes.
  arXiv  Detail & Related papers  (2023-07-02T21:14:49Z)
• Space Non-cooperative Object Active Tracking with Deep Reinforcement Learning [1.212848031108815]
  We propose an end-to-end active visual tracking method based on DQN algorithm, named as DRLAVT. It can guide the chasing spacecraft approach to arbitrary space non-cooperative target merely relied on color or RGBD images. It significantly outperforms position-based visual servoing baseline algorithm that adopts state-of-the-art 2D monocular tracker, SiamRPN.
  arXiv  Detail & Related papers  (2021-12-18T06:12:24Z)
• Polyline Based Generative Navigable Space Segmentation for Autonomous Visual Navigation [57.3062528453841]
  We propose a representation-learning-based framework to enable robots to learn the navigable space segmentation in an unsupervised manner. We show that the proposed PSV-Nets can learn the visual navigable space with high accuracy, even without any single label.
  arXiv  Detail & Related papers  (2021-10-29T19:50:48Z)
• Learning Perceptual Locomotion on Uneven Terrains using Sparse Visual Observations [75.60524561611008]
  This work aims to exploit the use of sparse visual observations to achieve perceptual locomotion over a range of commonly seen bumps, ramps, and stairs in human-centred environments. We first formulate the selection of minimal visual input that can represent the uneven surfaces of interest, and propose a learning framework that integrates such exteroceptive and proprioceptive data. We validate the learned policy in tasks that require omnidirectional walking over flat ground and forward locomotion over terrains with obstacles, showing a high success rate.
  arXiv  Detail & Related papers  (2021-09-28T20:25:10Z)
• Glimpse-Attend-and-Explore: Self-Attention for Active Visual Exploration [47.01485765231528]
  Active visual exploration aims to assist an agent with a limited field of view to understand its environment based on partial observations. We propose the Glimpse-Attend-and-Explore model which employs self-attention to guide the visual exploration instead of task-specific uncertainty maps. Our model provides encouraging results while being less dependent on dataset bias in driving the exploration.
  arXiv  Detail & Related papers  (2021-08-26T11:41:03Z)
• Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning [126.57680291438128]
  We study whether scalability can be achieved via a disentangled representation. We evaluate semantic tracklets' on the visual multi-agent particle environment (VMPE) and on the challenging visual multi-agent GFootball environment. Notably, this method is the first to successfully learn a strategy for five players in the GFootball environment using only visual data.
  arXiv  Detail & Related papers  (2021-08-06T22:19:09Z)
• ViNG: Learning Open-World Navigation with Visual Goals [82.84193221280216]
  We propose a learning-based navigation system for reaching visually indicated goals. We show that our system, which we call ViNG, outperforms previously-proposed methods for goal-conditioned reinforcement learning. We demonstrate ViNG on a number of real-world applications, such as last-mile delivery and warehouse inspection.
  arXiv  Detail & Related papers  (2020-12-17T18:22:32Z)
• Embodied Visual Active Learning for Semantic Segmentation [33.02424587900808]
  We study the task of embodied visual active learning, where an agent is set to explore a 3d environment with the goal to acquire visual scene understanding. We develop a battery of agents - both learnt and pre-specified - and with different levels of knowledge of the environment. We extensively evaluate the proposed models using the Matterport3D simulator and show that a fully learnt method outperforms comparable pre-specified counterparts.
  arXiv  Detail & Related papers  (2020-12-17T11:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.