Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers

• URL: http://arxiv.org/abs/2609.00358v1
• Date: Mon, 31 Aug 2026 20:51:51 GMT
• Title: Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers
• Abstract summary: Vision Transformers provide strong visual representations but typically rely on slowly updated parameters.<n>This work proposes textitHierarchical Hebbian Memory, a three-level memory architecture composed of rapid Working Memory, persistent Routed Episodic Memory, and slower Semantic Memory.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformers provide strong visual representations but typically rely on slowly updated parameters, limiting their ability to organize newly acquired information across different memory timescales. This work proposes \textit{Hierarchical Hebbian Memory}, a three-level memory architecture composed of rapid Working Memory, persistent Routed Episodic Memory, and slower Semantic Memory. A learned controller regulates memory contribution, read and write routing, plasticity, retention, and consolidation. A causal read-before-write lifecycle ensures that the current outcome cannot influence the prediction it supervises. The architecture is evaluated on Omniglot 5-way 1-shot recognition and CORe50 continual object recognition. With Swin-Tiny, the hierarchical model reaches 97.39\% accuracy on Omniglot and 95.37\% final accuracy on CORe50 when combined with experience replay. Learned multi-bank retrieval reaches 47.50\% delayed-association accuracy, compared with 24.17\% for a single persistent bank and 25.00\% without memory. After intervening distractors, Episodic Memory retains approximately 0.96 cosine similarity with stored associations, while Working Memory falls to approximately 0.05. These results show that Hebbian association and learned memory routing can jointly organize online visual experience across rapid, persistent, and consolidated memory timescales within Vision Transformers.

Related papers

• Remember Smarter: Visual History Compressor and Hyperbolic Experience Space for Robotic Memory [54.400138177707504]
  Remember Smarter (RS) is a plug-and-play module with complementary visual-history and hyperbolic experience-memory branches.<n>When adapted to pi0, RS increases total success on LIBERO-Plus from 53.6% to 70.6% and achieves substantial performance gains in real-robot experiments.
  arXiv  Detail & Related papers  (2026-08-15T15:13:33Z)
• TRUSTMEM: Learning Trustworthy Memory Consolidation for LLM Agents with Long-Term Memory [11.528637007309937]
  Large language model (LLM) agents rely on long-term memory to support extended interactions and personalized assistance beyond finite context windows.<n>Existing memory agents actively update external memory through generated write, revise, and delete operations.<n>These updates may omit important information, corrupt existing memory, or introduce unsupported hallucinated content.<n>We propose TrustMem, a framework designed to improve the trustworthiness of memory consolidation.
  arXiv  Detail & Related papers  (2026-06-23T20:49:28Z)
• MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery [58.64329475683699]
  We argue that long-term memory should be evaluated as an auditable post-interaction artifact.<n>We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users.<n>We find that successful assistance and recoverable memory behave as distinct capabilities.
  arXiv  Detail & Related papers  (2026-06-23T13:52:46Z)
• When Should Memory Stay Silent: Measuring Memory-Use Boundaries in Memory-Augmented Conversational Agents [13.354128891280316]
  Long-term memory enables language model agents to support personalized interactions.<n>It remains unclear when available memories warrant integration into responses.<n>Retrieval systems reduce exposure but do not eliminate integration once sensitive memory reaches the generator.
  arXiv  Detail & Related papers  (2026-06-04T11:54:51Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding [55.7992006853979]
  SAVEMem is a framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.<n>We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.
  arXiv  Detail & Related papers  (2026-05-08T15:40:40Z)
• LightMem: Lightweight and Efficient Memory-Augmented Generation [72.21680105265824]
  We introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems.<n>Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages.<n>Experiments on LongMemEval with GPT and Qwen backbones show that LightMem outperforms strong baselines in accuracy (up to 10.9% gains) while reducing token usage by up to 117x.
  arXiv  Detail & Related papers  (2025-10-21T17:58:17Z)
• MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation [59.31354761628506]
  Temporal context is essential for robotic manipulation because such tasks are inherently non-Markovian, yet mainstream VLA models typically overlook it.<n>We propose MemoryVLA, a Cognition-Memory-Action framework for long-horizon robotic manipulation.<n>We evaluate it on 150+ simulation and real-world tasks across three robots.
  arXiv  Detail & Related papers  (2025-08-26T17:57:16Z)
• HEMA : A Hippocampus-Inspired Extended Memory Architecture for Long-Context AI Conversations [0.0]
  Large language models (LLMs) struggle with maintaining coherence in extended conversations spanning hundreds of turns.<n>This paper introduces HEMA, a dual-memory system inspired by human cognitive processes.<n>With 10K indexed chunks, Vector Memory achieves P@5 >= 0.80 and R@50 >= 0.74, doubling the area under the precision-recall curve.
  arXiv  Detail & Related papers  (2025-04-23T14:27:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.