EgoMap: Projective mapping and structured egocentric memory for Deep RL

• URL: http://arxiv.org/abs/2002.02286v2
• Date: Fri, 7 Feb 2020 14:00:39 GMT
• Title: EgoMap: Projective mapping and structured egocentric memory for Deep RL
• Abstract summary: EgoMap is a spatially structured neural memory architecture. It augments a deep reinforcement learning agent's performance in 3D environments. We show it outperforms both standard recurrent agents and state of the art agents with structured memory.
• Score: 14.77454374243818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tasks involving localization, memorization and planning in partially observable 3D environments are an ongoing challenge in Deep Reinforcement Learning. We present EgoMap, a spatially structured neural memory architecture. EgoMap augments a deep reinforcement learning agent's performance in 3D environments on challenging tasks with multi-step objectives. The EgoMap architecture incorporates several inductive biases including a differentiable inverse projection of CNN feature vectors onto a top-down spatially structured map. The map is updated with ego-motion measurements through a differentiable affine transform. We show this architecture outperforms both standard recurrent agents and state of the art agents with structured memory. We demonstrate that incorporating these inductive biases into an agent's architecture allows for stable training with reward alone, circumventing the expense of acquiring and labelling expert trajectories. A detailed ablation study demonstrates the impact of key aspects of the architecture and through extensive qualitative analysis, we show how the agent exploits its structured internal memory to achieve higher performance.

Related papers

• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models [16.656416066183887]
  ArchSIBench is a Benchmark for Architectural Spatial Intelligence based on the perspectives from architecture, cognitive science, and psychology.<n> ArchSIBench covers five core dimensions: perception, reasoning, navigation, transformation, and configuration, comprising 17 fine-grained subtasks.<n>We evaluate various Vision-Language Models (VLMs) and find that the architectural spatial intelligence of most models shows significant differences from human baselines.
  arXiv  Detail & Related papers  (2026-05-20T07:27:08Z)
• Network of Theseus (like the ship) [17.044332179158392]
  Network of Theseus (NoT) is a method for converting a trained, or even untrained, guide network architecture part-by-part into an entirely different target network architecture.<n>At each stage, components in the guide network architecture are incrementally replaced with target architecture modules and aligned via representational similarity metrics.<n>NoT expands the space of viable inference-time architectures, opening opportunities for better accuracy-efficiency tradeoffs and enabling more directed exploration of the architectural design space.
  arXiv  Detail & Related papers  (2025-12-03T19:15:18Z)
• MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddings [5.647131476818603]
  A cognitive map is an internal model which encodes the abstract relationships among entities in the world.<n>We introduce MapFormers, new architectures based on Transformer models, which can learn cognitive maps from observational data.<n>MapFormers have broad applications in both neuroscience and AI, by explaining the neural mechanisms giving rise to cognitive maps.
  arXiv  Detail & Related papers  (2025-11-24T16:29:02Z)
• Localizing and Mitigating Memorization in Image Autoregressive Models [25.67535929806252]
  Image AutoRegressive (IAR) models have achieved state-of-the-art performance in speed and quality of generated images.<n>However, they also raise concerns about memorization of their training data and its implications for privacy.<n>This work explores where and how such memorization occurs within different image autoregressive architectures by measuring a fine-grained memorization.
  arXiv  Detail & Related papers  (2025-08-30T13:04:36Z)
• Attention-Guided Multi-Scale Local Reconstruction for Point Clouds via Masked Autoencoder Self-Supervised Learning [9.390627399366833]
  We introduce PointAMaLR, a novel self-supervised learning framework for point cloud processing.<n>PointAMaLR implements hierarchical reconstruction across multiple local regions.<n>Experiments on benchmark datasets demonstrate PointAMaLR's superior accuracy and quality in both classification and reconstruction tasks.
  arXiv  Detail & Related papers  (2025-07-05T16:17:49Z)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data [79.52833996220059]
  We present a unified framework for enhancing 3D spatial reasoning in pre-trained vision-language models without modifying their architecture.<n>This framework combines SpatialMind, a structured prompting strategy that decomposes complex scenes and questions into interpretable reasoning steps, with ScanForgeQA, a scalable question-answering dataset built from diverse 3D simulation scenes.
  arXiv  Detail & Related papers  (2025-06-04T07:36:33Z)
• Self-supervised Learning of Dense Hierarchical Representations for Medical Image Segmentation [2.2265038612930663]
  This paper demonstrates a self-supervised framework for learning voxel-wise coarse-to-fine representations tailored for dense downstream tasks. We devise a training strategy that balances the contributions of features from multiple scales, ensuring that the learned representations capture both coarse and fine-grained details.
  arXiv  Detail & Related papers  (2024-01-12T09:47:17Z)
• Multi-task Learning with 3D-Aware Regularization [55.97507478913053]
  We propose a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space. We show that the proposed method is architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance.
  arXiv  Detail & Related papers  (2023-10-02T08:49:56Z)
• Learning Navigational Visual Representations with Semantic Map Supervision [85.91625020847358]
  We propose a navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps. Ego$2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation.
  arXiv  Detail & Related papers  (2023-07-23T14:01:05Z)
• Learning the Space of Deep Models [12.785442275745973]
  We show how it is possible to use representation learning to learn a fixed-size, low-dimensional embedding space of trained deep models. We address image classification and neural representation of signals, showing how our embedding space can be learnt so as to capture the notions of performance and 3D shape.
  arXiv  Detail & Related papers  (2022-06-10T15:53:35Z)
• Improving Lidar-Based Semantic Segmentation of Top-View Grid Maps by Learning Features in Complementary Representations [3.0413873719021995]
  We introduce a novel way to predict semantic information from sparse, single-shot LiDAR measurements in the context of autonomous driving. The approach is aimed specifically at improving the semantic segmentation of top-view grid maps. For each representation a tailored deep learning architecture is developed to effectively extract semantic information.
  arXiv  Detail & Related papers  (2022-03-02T14:49:51Z)
• Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling [67.75095378830694]
  We develop a two-stage odometry estimation network, where we obtain the ego-motion by estimating a set of sub-region transformations. In this paper, we aim to alleviate the influence of unreliable structures in training, inference and mapping phases. Our two-frame odometry outperforms the previous state of the arts by 16%/12% in terms of translational/rotational errors.
  arXiv  Detail & Related papers  (2022-02-27T12:52:27Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)
• Occupancy Anticipation for Efficient Exploration and Navigation [97.17517060585875]
  We propose occupancy anticipation, where the agent uses its egocentric RGB-D observations to infer the occupancy state beyond the visible regions. By exploiting context in both the egocentric views and top-down maps our model successfully anticipates a broader map of the environment. Our approach is the winning entry in the 2020 Habitat PointNav Challenge.
  arXiv  Detail & Related papers  (2020-08-21T03:16:51Z)
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? [22.63641173256389]
  Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance.
  arXiv  Detail & Related papers  (2020-06-12T04:15:34Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)
• Semantically-Guided Representation Learning for Self-Supervised Monocular Depth [40.49380547487908]
  We propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.
  arXiv  Detail & Related papers  (2020-02-27T18:40:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.