Improving MLLMs in Embodied Exploration and Question Answering with Human-Inspired Memory Modeling

• URL: http://arxiv.org/abs/2602.15513v1
• Date: Tue, 17 Feb 2026 11:41:28 GMT
• Title: Improving MLLMs in Embodied Exploration and Question Answering with Human-Inspired Memory Modeling
• Abstract summary: We propose a non-parametric memory framework that disentangles episodic and semantic memory for embodied exploration and question answering.<n>Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning.
• Score: 7.901667874113712
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on textual summaries, which discard rich visual and spatial details and remain brittle in non-stationary environments. In this work, we propose a non-parametric memory framework that explicitly disentangles episodic and semantic memory for embodied exploration and question answering. Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning, enabling robust reuse of past observations without rigid geometric alignment. In parallel, we introduce a program-style rule extraction mechanism that converts experiences into structured, reusable semantic memory, facilitating cross-environment generalization. Extensive experiments demonstrate state-of-the-art performance on embodied question answering and exploration benchmarks, yielding a 7.3% gain in LLM-Match and an 11.4% gain in LLM MatchXSPL on A-EQA, as well as +7.7% success rate and +6.8% SPL on GOAT-Bench. Analyses reveal that our episodic memory primarily improves exploration efficiency, while semantic memory strengthens complex reasoning of embodied agents.

Related papers

• Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents [51.30250860677378]
  We propose MRAgent, a framework that combines an associative memory graph with an active reconstruction mechanism.<n>Running on a Cue-Tag-Content graph, our active reconstruction mechanism integrates reasoning directly into memory access.<n>Experiments on the LoCoMo benchmark and LongMemEval benchmark demonstrate significant improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T11:29:46Z)
• Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue [28.424707470781428]
  RefMem-Bench is a benchmark for reflective memory in long-horizon dialogue.<n>REMIND couples question-conditioned evidence retrieval, salience-aware grounding, and abstraction-level supervision.
  arXiv  Detail & Related papers  (2026-05-31T13:16:02Z)
• MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios [33.8882826707344]
  MemGround is a rigorous long-term memory benchmark grounded in rich, gamified interactive scenarios.<n>To comprehensively quantify both memory utilization and behavioral trajectories, we propose a multi-dimensional metric suite.
  arXiv  Detail & Related papers  (2026-03-23T02:57:39Z)
• Understanding LoRA as Knowledge Memory: An Empirical Analysis [20.53732426953178]
  This work investigates a parametric approach using Low-Rank Adaptation (LoRA) as a modular knowledge memory.<n>We bridge this gap through the first systematic empirical study mapping the design space of LoRA-based memory.<n>Our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.
  arXiv  Detail & Related papers  (2026-03-01T13:28:57Z)
• Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration [52.35887679314727]
  Long-term Memory Embodied Exploration aims to unify the agent's exploratory cognition and decision-making behaviors.<n>To enhance the agent's memory recall and proactive exploration capabilities, we propose MemoryExplorer.
  arXiv  Detail & Related papers  (2026-01-11T16:23:22Z)
• EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory [63.84216832544323]
  EvolMem is a new benchmark for assessing multi-session memory capabilities of large language models (LLMs) and agent systems.<n>To construct the benchmark, we introduce a hybrid data synthesis framework that consists of topic-initiated generation and narrative-inspired transformations.<n>Extensive evaluation reveals that no LLM consistently outperforms others across all memory dimensions.
  arXiv  Detail & Related papers  (2026-01-07T03:14:42Z)
• Memory in the Age of AI Agents [217.9368190980982]
  This work aims to provide an up-to-date landscape of current agent memory research.<n>We identify three dominant realizations of agent memory, namely token-level, parametric, and latent memory.<n>To support practical development, we compile a comprehensive summary of memory benchmarks and open-source frameworks.
  arXiv  Detail & Related papers  (2025-12-15T17:22:34Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)
• Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation [11.819481846962447]
  We investigate how agents built on pretrained large language models can learn target classification functions from labeled examples without parameter updates.<n>Our framework uses episodic memory to store instance-level critiques and distill these into reusable, task-level guidance.<n>Our findings highlight the promise of memory-driven, reflective learning for building more adaptive and interpretable LLM agents.
  arXiv  Detail & Related papers  (2025-10-22T17:58:03Z)
• Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation [8.14424628742372]
  Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions through environments.<n>Existing approaches for memory-persistent VLN face critical limitations.<n>We present Memoir, which employs imagination as a retrieval mechanism grounded by explicit memory.
  arXiv  Detail & Related papers  (2025-10-09T17:58:01Z)
• ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memory [21.4675019810992]
  Concept-level memory is reusable, modular abstractions distilled from solution traces and stored in natural language.<n>We evaluate on ARC-AGI, a benchmark that stresses compositional generalization and abstract reasoning.<n>We find abstract concepts to be the most consistent memory design, outscoring the baseline at all tested inference compute scales.
  arXiv  Detail & Related papers  (2025-09-04T17:54:19Z)
• In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents [70.12342024019044]
  Large Language Models (LLMs) have made significant progress in open-ended dialogue, yet their inability to retain and retrieve relevant information limits their effectiveness.<n>We propose Reflective Memory Management (RMM), a novel mechanism for long-term dialogue agents, integrating forward- and backward-looking reflections.<n>RMM shows more than 10% accuracy improvement over the baseline without memory management on the LongMemEval dataset.
  arXiv  Detail & Related papers  (2025-03-11T04:15:52Z)
• Episodic Memories Generation and Evaluation Benchmark for Large Language Models [7.660368798066376]
  We argue that integrating episodic memory capabilities into Large Language Models is essential for advancing AI towards human-like cognition.<n>We develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions.<n>We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance.
  arXiv  Detail & Related papers  (2025-01-21T02:16:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.