S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering

• URL: http://arxiv.org/abs/2605.28831v2
• Date: Mon, 08 Jun 2026 12:03:48 GMT
• Title: S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering
• Abstract summary: Long-horizon memory question answering often requires sparse evidence from heterogeneous histories.<n>We propose Structured Spatiotemporal Scene--Event Memory (S3Mem), a query-time memory interface that writes textual, visual, and agent-use histories into structured scene--event units and routes compact evidence packs to the reader.<n>Across LoCoMo, EMemBench Visual Games, and AMA-Bench, S3Mem provides a strong score--token trade-off, with the clearest gains on localized event, state, temporal, causal, or evidence provenance.
• Score: 58.90783999951707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon memory question answering often requires sparse evidence from heterogeneous histories, including events, object states, visual observations, temporal relations, and causal steps. Existing memory interfaces expand reader context, retrieve semantically related chunks, or expose graph neighborhoods, but they are not explicitly designed to select compact evidence for a fixed reader. We propose Structured Spatiotemporal Scene--Event Memory (S3Mem), a query-time memory interface that writes textual, visual, and agent-use histories into structured scene--event units and routes compact evidence packs to the reader. Its router scores candidate units, query anchors, and anchor--support links, enabling both single-hop selection and short multi-hop evidence chains without reader fine-tuning or test-time training. Across LoCoMo, EMemBench Visual Games, and AMA-Bench, S3Mem provides a strong score--token trade-off, with the clearest gains on localized event, state, temporal, causal, or provenance evidence. On LoCoMo, S3Mem reaches \(0.48\) F1 and \(0.40\) BLEU with (1{,}073) evidence tokens per question, about \(15.8\times\) fewer than the LoCoMo reference. On EMemBench Visual Games, it obtains the best F1 and second-best accuracy with only \(189\)tokens.On AMA-Bench, it is not the highest-scoring method, but remains competitive while using the fewest reader-visible evidence tokens.

Related papers

• CueMem: Cue-Guided Context Reconstruction for Long-Term Conversational Memory [38.76918926805768]
  CueMem is a cue-guided framework that treats extracted memory records as retrieval cues.<n>It reconstructs query-relevant dialogue context from their source turns.<n>Experiments on LoCoMo and LongMemEval show that CueMem consistently outperforms representative long-term memory baselines.
  arXiv  Detail & Related papers  (2026-09-11T02:27:22Z)
• MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents [24.912135848528845]
  Long-running LLM agents rely on external memory to store and reuse information beyond a single context window.<n>Key challenge in agent memory is not only to retrieve relevant records, but also to select necessary evidence under a given budget.<n>Existing memory readout methods mainly use textual or visual forms.
  arXiv  Detail & Related papers  (2026-09-07T13:24:40Z)
• EM^2Mem: Event-Centric Multimodal Memory for Large Language Models [52.3165591032625]
  We propose EM2Mem, an event-centric multimodal memory framework that aligns heterogeneous evidence to event anchors during memory construction.<n>Across three long-video QA benchmarks, EM2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points.
  arXiv  Detail & Related papers  (2026-09-01T01:38:41Z)
• Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents [3.6666917625778193]
  Agent Zero Memory is a provenance-aware long-term memory system.<n>It distils a user's conversations, files, and connected sources into three parallel memory systems.
  arXiv  Detail & Related papers  (2026-08-30T06:55:59Z)
• RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory [13.519130680921565]
  We introduce RippleMem, a long-term memory system that replaces one-shot retrieval with adaptive associative recollection.<n>LoCoMo and LongMemEval-S show that RippleMem achieves the best overall performance across evaluated settings.
  arXiv  Detail & Related papers  (2026-08-13T15:05:01Z)
• Narrative World Model: Narratology-Grounded Writer Memory for Long-Form Fiction [3.6314288564136064]
  General-purpose retrieval and agent-memory systems represent entities and facts but not the narratological structure these questions turn on.<n>We introduce the Narrative World Model (NWM), a writer-memory system that pairs a narratology-grounded typed temporal-state graph with query-conditioned hybrid retrieval.
  arXiv  Detail & Related papers  (2026-07-06T19:23:52Z)
• Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning [77.7531245219403]
  textscHomer is a Hierarchical Online Memory Exploration and Reasoning framework.<n>textscHomer's memory mirrors the multi-scale structure of long videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations.<n>textscHomer outperforms the previous best agent method by $+5.5$, $+10.8$, and $+4.4$ points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long.
  arXiv  Detail & Related papers  (2026-07-01T05:53:09Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory [80.97855900579512]
  Existing evaluations rarely test whether agents preserve the visual evidence needed for later reasoning.<n>We introduce MemEye, a framework that evaluates memory capabilities from two dimensions.<n>Under this framework, we construct a new benchmark across 8 life-scenario tasks.
  arXiv  Detail & Related papers  (2026-05-14T17:37:52Z)
• AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents [49.63422082885992]
  We propose AdaMem, an adaptive user-centric memory framework for long-horizon dialogue agents.<n>AdaMem organizes dialogue history into working, episodic, persona, and graph memories.<n>We evaluate AdaMem on the LoCoMo and PERSONAMEM benchmarks for long-horizon reasoning and user modeling.
  arXiv  Detail & Related papers  (2026-03-17T13:22:54Z)
• TraceMem: Weaving Narrative Memory Schemata from User Conversational Traces [9.654990538033362]
  Sustaining long-term interactions remains a bottleneck for Large Language Models.<n>We propose TraceMem, a framework that weaves structured, narrative memory schemata from user conversational traces.<n>TraceMem achieves state-of-the-art performance with a brain-inspired architecture.
  arXiv  Detail & Related papers  (2026-02-10T12:14:58Z)
• EverMemBench: Benchmarking Long-Term Interactive Memory in Large Language Models [16.865998112859604]
  We introduce EverMemBench, a benchmark featuring multi-party, multi-group conversations spanning over 1 million tokens.<n>EverMemBench evaluates memory systems across three dimensions through 1,000+ QA pairs.
  arXiv  Detail & Related papers  (2026-02-01T16:13:08Z)
• EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents [52.567469286881426]
  We introduce EMemBench, a programmatic benchmark for evaluating long-term memory of agents through interactive games.<n>Rather than using a fixed set of questions, EMemBench generates questions from each agent's own trajectory.<n>Each template computes verifiable ground truth from underlying game signals.
  arXiv  Detail & Related papers  (2026-01-23T12:09:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.