Do You Remember? Toward Memory-Centric Multimodal AI

• URL: http://arxiv.org/abs/2607.11919v1
• Date: Tue, 07 Jul 2026 19:07:27 GMT
• Title: Do You Remember? Toward Memory-Centric Multimodal AI
• Abstract summary: We present DoYouRemember, a three-stage architecture introducing reconstructive memory into MLLMs.<n>On 1,000 3D facial skin texture maps and 99,000 unlabeled facial images, we find that LLM hidden states contain approximately zero recoverable visual information.<n>We unify these findings under an information-theoretic framework: memory is lossy compression, recall is decompression, and hallucination is an inherent property of lossy decompression rather than a defect.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human memory is reconstructive, not a faithful recording. Current multimodal LLMs (MLLMs) lack this capability: they process images through a frozen visual encoder, produce a one-shot text output, and discard internal representations. We present DoYouRemember, a three-stage architecture introducing reconstructive memory into MLLMs: (1) a VQ-VAE compresses images into discrete visual tokens, (2) a LoRA-fine-tuned LLM jointly attends to visual and text tokens, and (3) a Diffusion Decoder reconstructs images from the LLM's hidden states. On 1,000 3D facial skin texture maps and 99,000 unlabeled facial images, we find that LLM hidden states contain approximately zero recoverable visual information -- the same Decoder producing clear reconstructions from VQ-VAE tokens (pre-LLM) produces pure noise from LLM hidden states (post-LLM), demonstrating that the LLM understands images but does not remember them. Training a shared memory matrix M under backpropagation systematically fails due to gradient cancellation (O(1/sqrt(N)) attenuation). We identify three root causes and show that local EMA updating resolves all three: each image updates only its top-8 slots out of 64, preserving inter-slot diversity. The resulting M (229K parameters, 16x compressed) approaches the VQ upper bound on unseen test images. Scaling to 1,024 slots surpasses it (LPIPS 0.056 vs. 0.071), as M's continuous representation avoids VQ quantization error. We unify these findings under an information-theoretic framework: memory is lossy compression, recall is decompression, and hallucination is an inherent property of lossy decompression rather than a defect.

Related papers

• MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory [12.853875632489114]
  When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success.<n>We find that the fact can remain recoverable from retained user images, including images tagged to entirely different facts.<n>We introduce the Information Provenance Graph (IPG), a taxonomy that classifies memory representations by deletion affordance.
  arXiv  Detail & Related papers  (2026-06-29T05:07:02Z)
• Image Prompt Reconstruction Attacks on Distributed MLLM Inference Frameworks [21.9083862179787]
  We investigate privacy risks to input images caused by intermediate embeddings in distributed MLLM frameworks.<n>We develop two passive black-box image reconstruction attacks, MPAA and IEDA, reflecting realistic threats from normal participants.<n>Results show consistently superior reconstruction performance in various settings.
  arXiv  Detail & Related papers  (2026-06-17T05:51:14Z)
• Echo-Memory: A Controlled Study of Memory in Action World Models [58.77172260403133]
  We present a controlled study of memory mechanisms in action-conditioned world models.<n>Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences.<n>We use Echo-Memory to fix the action-to-video interface and varies only how history is stored and read by the generator.
  arXiv  Detail & Related papers  (2026-06-08T17:54:10Z)
• MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models [50.25006399944962]
  Memory is essential for large vision-language models to handle long, multimodal interactions.<n>MEMLENS is a benchmark for memory in multimodal multi-session conversations.<n>We evaluate 27 LVLMs and 7 memory-augmented agents.
  arXiv  Detail & Related papers  (2026-05-14T14:41:17Z)
• From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents [78.30630000529133]
  We propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory.<n> MM-Mem memory structures hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic.<n>Experiments confirm the effectiveness of MM-Mem on both offline and streaming tasks.
  arXiv  Detail & Related papers  (2026-03-02T05:12:45Z)
• Agentic Learner with Grow-and-Refine Multimodal Semantic Memory [50.81667005063605]
  ViLoMem is a dual-stream memory framework that constructs compact, schema-based memory.<n>It encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
  arXiv  Detail & Related papers  (2025-11-26T18:55:08Z)
• CoMemo: LVLMs Need Image Context with Image Memory [51.681858871027345]
  CoMemo is a dual-path architecture that combines a Context image path with an image Memory path for visual processing.<n>We introduce RoPE-DHR, a novel positional encoding mechanism that employs thumbnail-based positional aggregation to maintain 2D spatial awareness.
  arXiv  Detail & Related papers  (2025-06-06T17:59:06Z)
• PIP-MM: Pre-Integrating Prompt Information into Visual Encoding via Existing MLLM Structures [5.513631883813244]
  We propose a framework that textbfPre-textbfIntegratestextbfPrompt information into the visual encoding process using existingmodules of MLLMs. Our model maintains excellent generation even when half of the visual tokens are reduced.
  arXiv  Detail & Related papers  (2024-10-30T15:05:17Z)
• Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models [26.32657568461926]
  multimodal large language models (MLLMs) are prone to hallucinations.<n>MemVR is a novel decoding paradigm inspired by common cognition.<n>MemVR significantly mitigates hallucination across various MLLMs.
  arXiv  Detail & Related papers  (2024-10-04T16:30:54Z)
• Auto-Encoding Morph-Tokens for Multimodal LLM [151.2618346912529]
  We propose encoding images into morph-tokens to serve a dual purpose: for comprehension, they act as visual prompts instructing MLLM to generate texts. Experiments show that morph-tokens can achieve a new SOTA for multimodal comprehension and generation simultaneously.
  arXiv  Detail & Related papers  (2024-05-03T08:43:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.