Memory-Centric Embodied Question Answer

• URL: http://arxiv.org/abs/2505.13948v1
• Date: Tue, 20 May 2025 05:27:57 GMT
• Title: Memory-Centric Embodied Question Answer
• Authors: Mingliang Zhai, Zhi Gao, Yuwei Wu, Yunde Jia,
• Abstract summary: Embodied Question Answering (EQA) requires agents to autonomously explore and understand the environment to answer context-dependent questions.<n>We propose a memory-centric EQA framework named MemoryEQA.<n>Unlike planner-centric EQA models where the memory module cannot fully interact with other modules, MemoryEQA flexible feeds memory information into all modules.
• Score: 39.3863762723862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied Question Answering (EQA) requires agents to autonomously explore and understand the environment to answer context-dependent questions. Existing frameworks typically center around the planner, which guides the stopping module, memory module, and answering module for reasoning. In this paper, we propose a memory-centric EQA framework named MemoryEQA. Unlike planner-centric EQA models where the memory module cannot fully interact with other modules, MemoryEQA flexible feeds memory information into all modules, thereby enhancing efficiency and accuracy in handling complex tasks, such as those involving multiple targets across different regions. Specifically, we establish a multi-modal hierarchical memory mechanism, which is divided into global memory that stores language-enhanced scene maps, and local memory that retains historical observations and state information. When performing EQA tasks, the multi-modal large language model is leveraged to convert memory information into the required input formats for injection into different modules. To evaluate EQA models' memory capabilities, we constructed the MT-HM3D dataset based on HM3D, comprising 1,587 question-answer pairs involving multiple targets across various regions, which requires agents to maintain memory of exploration-acquired target information. Experimental results on HM-EQA, MT-HM3D, and OpenEQA demonstrate the effectiveness of our framework, where a 19.8% performance gain on MT-HM3D compared to baseline model further underscores memory capability's pivotal role in resolving complex tasks.

Related papers

• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents [84.62985963113245]
  We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with constant memory across long multi-turn tasks.<n>At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning.<n>We show that MEM1-7B improves performance by 3.5x while reducing memory usage by 3.7x compared to Qwen2.5-14B-Instruct on a 16-objective multi-hop QA task.
  arXiv  Detail & Related papers  (2025-06-18T19:44:46Z)
• Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions [55.19217798774033]
  Memory is a fundamental component of AI systems, underpinning large language models (LLMs) based agents.<n>We introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression.<n>This survey provides a structured and dynamic perspective on research, benchmark datasets, and tools related to memory in AI.
  arXiv  Detail & Related papers  (2025-05-01T17:31:33Z)
• LM2: Large Memory Models [11.320069795732058]
  This paper introduces the Large Memory Model (LM2), a decoder-only Transformer architecture enhanced with an auxiliary memory module.<n> Experimental results on the BABILong benchmark demonstrate that the LM2model outperforms both the memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3% on average across tasks.
  arXiv  Detail & Related papers  (2025-02-09T22:11:42Z)
• A Framework for Inference Inspired by Human Memory Mechanisms [9.408704431898279]
  We propose a PMI framework that consists of perception, memory and inference components. The memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience. We apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets.
  arXiv  Detail & Related papers  (2023-10-01T08:12:55Z)
• SCM: Enhancing Large Language Model with Self-Controlled Memory Framework [54.33686574304374]
  Large Language Models (LLMs) are constrained by their inability to process lengthy inputs, resulting in the loss of critical historical information.<n>We propose the Self-Controlled Memory (SCM) framework to enhance the ability of LLMs to maintain long-term memory and recall relevant information.
  arXiv  Detail & Related papers  (2023-04-26T07:25:31Z)
• MeMOT: Multi-Object Tracking with Memory [97.48960039220823]
  Our model, called MeMOT, consists of three main modules that are all Transformer-based. MeMOT observes very competitive performance on widely adopted MOT datasets.
  arXiv  Detail & Related papers  (2022-03-31T02:33:20Z)
• Memory-Guided Semantic Learning Network for Temporal Sentence Grounding [55.31041933103645]
  We propose a memory-augmented network that learns and memorizes the rarely appeared content in TSG tasks. MGSL-Net consists of three main parts: a cross-modal inter-action module, a memory augmentation module, and a heterogeneous attention module.
  arXiv  Detail & Related papers  (2022-01-03T02:32:06Z)
• End-to-End Egospheric Spatial Memory [32.42361470456194]
  We propose a parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent. ESM can be trained end-to-end via either imitation or reinforcement learning. We show applications to semantic segmentation on the ScanNet dataset, where ESM naturally combines image-level and map-level inference modalities.
  arXiv  Detail & Related papers  (2021-02-15T18:59:07Z)
• Memorizing Comprehensively to Learn Adaptively: Unsupervised Cross-Domain Person Re-ID with Multi-level Memory [89.43986007948772]
  We propose a novel multi-level memory network (MMN) to discover multi-level complementary information in the target domain. Unlike the simple memory in previous works, we propose a novel multi-level memory network (MMN) to discover multi-level complementary information in the target domain.
  arXiv  Detail & Related papers  (2020-01-13T09:48:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.