Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving

• URL: http://arxiv.org/abs/2609.08217v1
• Date: Tue, 08 Sep 2026 03:59:10 GMT
• Title: Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving
• Abstract summary: VLMs have shown promise for autonomous driving yet still suffer from weak-temporal perception, and limited generalization.<n>Recent methods improve reasoning and decision-making through CoT explanations, retrieval-augmented generation or the static injection of tool outputs.<n>We present to our knowledge, the first synergistic framework that tightly couples hierarchical memory with proactive tool invocation in a closed loop.
• Score: 0.8419406375517101
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: VLMs have shown promise for autonomous driving, yet still suffer from hallucination, weak spatio-temporal perception, and limited generalization. Recent methods improve reasoning and decision-making through CoT explanations, retrieval-augmented generation or the static injection of tool outputs. Although these mechanisms enrich the context, the model neither proactively perceives scene information nor accumulates experience after answering. To overcome these limitations, we present, to our knowledge, the first synergistic framework that tightly couples hierarchical memory with proactive tool invocation in a closed reasoning loop. Our contributions are threefold. (i) Hierarchical Driving Memory: a scene-level short-term memory maintains the dynamic scene state, and an evolving long-term memory retrieves reusable experience and tool strategies. (ii) Memory-Tool Synergistic Reasoning Framework: guided by the scene state and retrieved experience, the model adaptively invokes tools to refine its reasoning at inference time and consolidates reusable experience into a long-term memory pool offline. (iii) Data Generation and Two-stage Training Pipeline: verified memory-tool trajectories built by multi-step teacher rollout are used to train with SFT and GRPO. Our 7B model reaches an overall reasoning score of 80.03 and MCQ accuracy of 79.09% on DriveLMM-o1, surpassing the strongest baseline by 7.74 MCQ points and generalizes strongly across benchmarks. Notably, ablation and analysis studies validate the effectiveness of each component and further reveal the complementary roles of hierarchical memory. Short-term memory strengthens spatio-temporal understanding, improving STSBench accuracy by 24.2 points, while offline long-term memory consolidation yields an additional 3.57-point MCQ gain with all parameters frozen, demonstrating continual self-evolution through accumulated driving experience.

Related papers

• Retrieval-Driven Memory Reconsolidation for Long-Term LLM Agents [39.55670043123564]
  Existing memory systems update memory when new information arrives.<n>Retrieval feedback is rarely exploited to reorganize memory for future access continuously.<n>We propose textbfREALM, a framework for continually evolving long-term memory in LLM agents.
  arXiv  Detail & Related papers  (2026-09-13T05:17:37Z)
• Dual-Latent Memory Routing for Vision-Language Reasoning [39.90357280328134]
  Multimodal large language models (MLLMs) have recently made strong progress in vision-language reasoning, yet their performance often degrades as generations grow longer.<n>Inspired by how humans separately recall what they see and what they infer when solving complex tasks, we propose DLMR.<n>It equips MLLMs with Dual Latent Memories: a visual memory that compresses image evidence and a reasoning memory that tracks intermediate conclusions and constraints.
  arXiv  Detail & Related papers  (2026-09-02T10:45:25Z)
• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents [97.19033877543522]
  We present a controlled harness evaluation of memory substrates for memory-augmented agents.<n>We instrument 26 performance and efficiency metrics under a unified harness.<n>These findings motivate substrate routing as a necessary component of adaptive agent memory systems.
  arXiv  Detail & Related papers  (2026-08-15T03:29:48Z)
• S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval [38.608941713774186]
  We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses.<n>S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream.
  arXiv  Detail & Related papers  (2026-07-02T18:29:20Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• MA-DLE: Speech-based Automatic Depression Level Estimation via Memory Augmentation [59.709198999959206]
  Speech-based automatic estimation of depression levels is essential for enabling early detection and timely intervention.<n>Most existing approaches rely on RNN-based architectures to model temporal information for depression estimation.<n>We introduce a memory-based feature augmentation method that enhances the representational capacity of GRU-extracted features.
  arXiv  Detail & Related papers  (2026-04-20T14:42:13Z)
• LightThinker++: From Reasoning Compression to Memory Management [61.2260619973687]
  Large language models (LLMs) excel at complex reasoning, yet their efficiency is limited by the surging cognitive overhead of long thought traces.<n>We propose LightThinker, a method that enables LLMs to dynamically compress intermediate thoughts into compact semantic representations.<n>We evolve the framework into LightThinker++, introducing Explicit Adaptive Memory Management.
  arXiv  Detail & Related papers  (2026-04-04T10:46:09Z)
• MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios [33.8882826707344]
  MemGround is a rigorous long-term memory benchmark grounded in rich, gamified interactive scenarios.<n>To comprehensively quantify both memory utilization and behavioral trajectories, we propose a multi-dimensional metric suite.
  arXiv  Detail & Related papers  (2026-03-23T02:57:39Z)
• From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents [78.30630000529133]
  We propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory.<n> MM-Mem memory structures hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic.<n>Experiments confirm the effectiveness of MM-Mem on both offline and streaming tasks.
  arXiv  Detail & Related papers  (2026-03-02T05:12:45Z)
• Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation [8.14424628742372]
  Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions through environments.<n>Existing approaches for memory-persistent VLN face critical limitations.<n>We present Memoir, which employs imagination as a retrieval mechanism grounded by explicit memory.
  arXiv  Detail & Related papers  (2025-10-09T17:58:01Z)
• MoSFormer: Augmenting Temporal Context with Memory of Surgery for Surgical Phase Recognition [6.913838841605972]
  Memory of Surgery (MoS) is a framework that enriches temporal modeling by incorporating both semantic interpretable long-term surgical history and short-term impressions.<n>MoSFormer demonstrates state-of-the-art performance on multiple benchmarks.
  arXiv  Detail & Related papers  (2025-03-02T02:26:21Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.