Automated mapping of virtual environments with visual predictive coding

• URL: http://arxiv.org/abs/2308.10913v2
• Date: Wed, 17 Apr 2024 23:27:02 GMT
• Title: Automated mapping of virtual environments with visual predictive coding
• Authors: James Gornet, Matthew Thomson,
• Abstract summary: We introduce a framework in which an agent navigates a virtual environment while engaging in visual predictive coding. While learning a next image prediction task, the agent automatically constructs an internal representation of the environment that quantitatively reflects distances. The internal map enables the agent to pinpoint its location relative to landmarks using only visual information.
• Score: 0.9591674293850556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans construct internal cognitive maps of their environment directly from sensory inputs without access to a system of explicit coordinates or distance measurements. While machine learning algorithms like SLAM utilize specialized visual inference procedures to identify visual features and construct spatial maps from visual and odometry data, the general nature of cognitive maps in the brain suggests a unified mapping algorithmic strategy that can generalize to auditory, tactile, and linguistic inputs. Here, we demonstrate that predictive coding provides a natural and versatile neural network algorithm for constructing spatial maps using sensory data. We introduce a framework in which an agent navigates a virtual environment while engaging in visual predictive coding using a self-attention-equipped convolutional neural network. While learning a next image prediction task, the agent automatically constructs an internal representation of the environment that quantitatively reflects distances. The internal map enables the agent to pinpoint its location relative to landmarks using only visual information.The predictive coding network generates a vectorized encoding of the environment that supports vector navigation where individual latent space units delineate localized, overlapping neighborhoods in the environment. Broadly, our work introduces predictive coding as a unified algorithmic framework for constructing cognitive maps that can naturally extend to the mapping of auditory, sensorimotor, and linguistic inputs.

Related papers

• FindAnything: Open-Vocabulary and Object-Centric Mapping for Robot Exploration in Any Environment [16.987872206495897]
  FindAnything is an open-world mapping framework that incorporates vision-language information into dense volumetric submaps. Our system is the first of its kind to be deployed on resource-constrained devices, such as MAVs.
  arXiv  Detail & Related papers  (2025-04-11T15:12:05Z)
• Meta-Representational Predictive Coding: Biomimetic Self-Supervised Learning [51.22185316175418]
  We present a new form of predictive coding that we call meta-representational predictive coding (MPC) MPC sidesteps the need for learning a generative model of sensory input by learning to predict representations of sensory input across parallel streams.
  arXiv  Detail & Related papers  (2025-03-22T22:13:14Z)
• Exploring Emerging Trends and Research Opportunities in Visual Place Recognition [28.76562316749074]
  Visual-based recognition is a long-standing challenge in computer vision and robotics communities. Visual place recognition is vital for most localization implementations. Researchers have recently turned their attention to vision-language models.
  arXiv  Detail & Related papers  (2024-11-18T11:36:17Z)
• Mapping High-level Semantic Regions in Indoor Environments without Object Recognition [50.624970503498226]
  The present work proposes a method for semantic region mapping via embodied navigation in indoor environments. To enable region identification, the method uses a vision-to-language model to provide scene information for mapping. By projecting egocentric scene understanding into the global frame, the proposed method generates a semantic map as a distribution over possible region labels at each location.
  arXiv  Detail & Related papers  (2024-03-11T18:09:50Z)
• Finding Concept Representations in Neural Networks with Self-Organizing Maps [2.817412580574242]
  We show how self-organizing maps can be used to inspect how activation of layers of neural networks correspond to neural representations of abstract concepts. We show that, among the measures tested, the relative entropy of the activation map for a concept is a suitable candidate and can be used as part of a methodology to identify and locate the neural representation of a concept.
  arXiv  Detail & Related papers  (2023-12-10T12:10:34Z)
• Active Neural Mapping [20.242598287146578]
  We address the problem of active mapping with a continually-learned neural scene representation, namely Active Neural Mapping. We present for the first time an active mapping system with a coordinate-based implicit neural representation for online scene reconstruction.
  arXiv  Detail & Related papers  (2023-08-30T18:07:30Z)
• Learning Navigational Visual Representations with Semantic Map Supervision [85.91625020847358]
  We propose a navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps. Ego$2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation.
  arXiv  Detail & Related papers  (2023-07-23T14:01:05Z)
• Neural Implicit Dense Semantic SLAM [83.04331351572277]
  We propose a novel RGBD vSLAM algorithm that learns a memory-efficient, dense 3D geometry, and semantic segmentation of an indoor scene in an online manner. Our pipeline combines classical 3D vision-based tracking and loop closing with neural fields-based mapping. Our proposed algorithm can greatly enhance scene perception and assist with a range of robot control problems.
  arXiv  Detail & Related papers  (2023-04-27T23:03:52Z)
• Multi-Object Navigation with dynamically learned neural implicit representations [10.182418917501064]
  We propose to structure neural networks with two neural implicit representations, which are learned dynamically during each episode. We evaluate the agent on Multi-Object Navigation and show the high impact of using neural implicit representations as a memory source.
  arXiv  Detail & Related papers  (2022-10-11T04:06:34Z)
• Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams [64.82800502603138]
  This paper proposes a novel neural-network-based approach to progressively and autonomously develop pixel-wise representations in a video stream. The proposed method is based on a human-like attention mechanism that allows the agent to learn by observing what is moving in the attended locations. Our experiments leverage 3D virtual environments and they show that the proposed agents can learn to distinguish objects just by observing the video stream.
  arXiv  Detail & Related papers  (2022-04-26T09:52:31Z)
• Neural Network based Successor Representations of Space and Language [6.748976209131109]
  We present a neural network based approach to learn multi-scale successor representations of structured knowledge. In all scenarios, the neural network correctly learns and approximates the underlying structure by building successor representations. We conclude that cognitive maps and neural network-based successor representations of structured knowledge provide a promising way to overcome some of the short comings of deep learning towards artificial general intelligence.
  arXiv  Detail & Related papers  (2022-02-22T21:52:46Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Neural Topological SLAM for Visual Navigation [112.73876869904]
  We design topological representations for space that leverage semantics and afford approximate geometric reasoning. We describe supervised learning-based algorithms that can build, maintain and use such representations under noisy actuation.
  arXiv  Detail & Related papers  (2020-05-25T17:56:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.