STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memory

• URL: http://arxiv.org/abs/2602.09255v2
• Date: Thu, 12 Feb 2026 06:19:54 GMT
• Title: STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memory
• Abstract summary: Mobile robots are often deployed over long durations in diverse open, dynamic scenes.<n>A core challenge is to build a scalable long-horizon memory that supports an agentic workflow for planning, retrieval, and reasoning.<n>We present STaR, a task-agnostic, multimodal long-term memory that generalizes to unseen queries.
• Score: 16.76678779219906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mobile robots are often deployed over long durations in diverse open, dynamic scenes, including indoor setting such as warehouses and manufacturing facilities, and outdoor settings such as agricultural and roadway operations. A core challenge is to build a scalable long-horizon memory that supports an agentic workflow for planning, retrieval, and reasoning over open-ended instructions at variable granularity, while producing precise, actionable answers for navigation. We present STaR, an agentic reasoning framework that (i) constructs a task-agnostic, multimodal long-term memory that generalizes to unseen queries while preserving fine-grained environmental semantics (object attributes, spatial relations, and dynamic events), and (ii) introduces a Scalable Task Conditioned Retrieval algorithm based on the Information Bottleneck principle to extract from long-term memory a compact, non-redundant, information-rich set of candidate memories for contextual reasoning. We evaluate STaR on NaVQA (mixed indoor/outdoor campus scenes) and WH-VQA, a customized warehouse benchmark with many visually similar objects built with Isaac Sim, emphasizing contextual reasoning. Across the two datasets, STaR consistently outperforms strong baselines, achieving higher success rates and markedly lower spatial error. We further deploy STaR on a real Husky wheeled robot in both indoor and outdoor environments, demonstrating robust long horizon reasoning, scalability, and practical utility. Project Website: https://trailab.github.io/STaR-website/

Related papers

• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memory [10.92726806758712]
  RAVEN is an agentic memory system for long-horizon robotic question answering and navigation.<n>RAVEN stores visual embeddings with pose and time in a vector database, and grounds retrieval in a spatial map to answer queries.<n>We instantiate RAVEN on a Unitree Go1 robot for the task of long-horizon navigation for natural language goal-reaching.
  arXiv  Detail & Related papers  (2026-06-23T21:56:41Z)
• RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark [49.16825786282095]
  Memory is a critical component of robotic intelligence, as robots must rely on past observations and actions to accomplish long-horizon tasks.<n>Existing robotic memory benchmarks still lack multimodal annotations for memory formation, provide limited task coverage and structural complexity, and remain restricted to simulation without real-world evaluation.<n>We address this gap with RoboMemArena, a large-scale benchmark of 26 tasks, with average trajectory lengths exceeding 1,000 steps per task and 68.9% of subtasks being memory-dependent.
  arXiv  Detail & Related papers  (2026-05-11T17:54:49Z)
• Beyond Short-Horizon: VQ-Memory for Robust Long-Horizon Manipulation in Non-Markovian Simulation Benchmarks [96.60530830276281]
  RuleSafe is a new articulated manipulation benchmark built upon a scalable LLM-aided simulation framework.<n>VQ-Memory is a compact and structured temporal representation that uses vector-quantized variational autoencoders.
  arXiv  Detail & Related papers  (2026-03-10T11:13:54Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation [56.30931340537373]
  MolmoSpaces is a fully open ecosystem to support benchmarking of robot policies.<n>MolmoSpaces consists of over 230k diverse indoor environments.<n>MolmoSpaces-Bench is a benchmark suite of 8 tasks in which robots interact with our diverse scenes and richly annotated objects.
  arXiv  Detail & Related papers  (2026-02-11T20:16:31Z)
• AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts [78.33143446024485]
  We introduce textbfAgentLongBench, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles.<n>This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios.
  arXiv  Detail & Related papers  (2026-01-28T16:05:44Z)
• Beyond Fact Retrieval: Episodic Memory for RAG with Generative Semantic Workspaces [5.110309385104824]
  Large Language Models (LLMs) face fundamental challenges in long-context reasoning.<n>Current solutions fail to build the space-time-anchored narrative representations required for tracking entities through episodic events.<n>We propose the textbfGenerative Semantic Workspace (GSW), a neuro-inspired generative memory framework.
  arXiv  Detail & Related papers  (2025-11-10T19:59:34Z)
• Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoning [5.740131013400576]
  We propose Meta-Memory, a large language model (LLM)-driven agent that constructs a high-density memory representation of the environment.<n>The key innovation of Meta-Memory lies in its capacity to retrieve and integrate relevant memories through joint reasoning over semantic and spatial modalities.<n> Experimental results show that Meta-Memory significantly outperforms state-of-the-art methods on both the SpaceLocQA and the public NaVQA benchmarks.
  arXiv  Detail & Related papers  (2025-09-25T05:22:52Z)
• MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents [84.62985963113245]
  We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with constant memory across long multi-turn tasks.<n>At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning.<n>We show that MEM1-7B improves performance by 3.5x while reducing memory usage by 3.7x compared to Qwen2.5-14B-Instruct on a 16-objective multi-hop QA task.
  arXiv  Detail & Related papers  (2025-06-18T19:44:46Z)
• FindingDory: A Benchmark to Evaluate Memory in Embodied Agents [49.18498389833308]
  We introduce a new benchmark for long-range embodied tasks in the Habitat simulator.<n>This benchmark evaluates memory-based capabilities across 60 tasks requiring sustained engagement and contextual awareness.
  arXiv  Detail & Related papers  (2025-06-18T17:06:28Z)
• Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation [69.01029651113386]
  Embodied-RAG is a framework that enhances the model of an embodied agent with a non-parametric memory system.<n>At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail.<n>We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 250 explanation and navigation queries.
  arXiv  Detail & Related papers  (2024-09-26T21:44:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.