S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval

• URL: http://arxiv.org/abs/2607.02689v1
• Date: Thu, 02 Jul 2026 18:29:20 GMT
• Title: S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval
• Abstract summary: We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses.<n>S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream.
• Score: 38.608941713774186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As wearable devices enable continuous first-person recording, AI assistants must reason across long time horizons to recall past experiences-a capability known as episodic memory. Current benchmarks often rely on offline evaluation with access to entire video files, failing to simulate the streaming reality of wearable intelligence. We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses. S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream. We provide 9,448 QA pairs requiring manual visual proof through precise temporal localization and supporting flexible response lengths to simulate natural human-AI interaction. Our extensive benchmarking of frontier models uncovers a localization paradox: while semantic reasoning improves with parameter scale, temporal grounding precision remains a stagnant architectural bottleneck that does not benefit from brute-force increases in model size, resolution, or frame density. S-EMBER establishes a hardware-authentic foundation for developing grounded, reliable episodic memory in the next generation of wearable AI agents.

Related papers

• Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving [0.8419406375517101]
  VLMs have shown promise for autonomous driving yet still suffer from weak-temporal perception, and limited generalization.<n>Recent methods improve reasoning and decision-making through CoT explanations, retrieval-augmented generation or the static injection of tool outputs.<n>We present to our knowledge, the first synergistic framework that tightly couples hierarchical memory with proactive tool invocation in a closed loop.
  arXiv  Detail & Related papers  (2026-09-08T03:59:10Z)
• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams [58.77207336324662]
  UCS-Bench is a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions.<n>We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations.
  arXiv  Detail & Related papers  (2026-06-13T08:50:49Z)
• Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation [59.01737077739414]
  We present Echo Infinity, an autoregressive (AR) framework towards real-time infinite video generation.<n>It employs a learnable evolving memory to dynamically filter, abstract, and compress any-length history at constant cost.<n>It achieves state-of-the-art performance, and, to our knowledge, demonstrates promising 24-hour (>1.3 M frames) real-time rollouts for the first time.
  arXiv  Detail & Related papers  (2026-06-03T07:09:01Z)
• SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory [14.96864764261795]
  SuperMemory-VQA is a dataset for evaluating AI assistants on practical, long-horizon memory tasks.<n>It contains 52.9 hours of everyday activities recorded with AI glasses, including synchronized RGB video, audio transcription, eye gaze, IMU, and SLAM trajectories.<n>We construct grounded 4,853 question-answer pairs that span object and location memory, intent recall, visual scene recall, timeline reconstruction, conversational memory, and in-context retrieval.
  arXiv  Detail & Related papers  (2026-05-30T17:53:10Z)
• EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision [9.701124246177661]
  Continuous episodic memory is a core capability for autonomous agents.<n>Egostream is a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision.
  arXiv  Detail & Related papers  (2026-05-29T17:20:10Z)
• EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding [89.26501160264199]
  EgoMemReason systematically evaluates week-long egocentric video understanding through memory-driven reasoning.<n>EgoMemReason comprises 500 questions across three memory types and six core challenges.<n>We evaluate EgoMemReason on 17 methods across MLLMs and agentic frameworks.
  arXiv  Detail & Related papers  (2026-05-11T01:59:59Z)
• AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents [49.63422082885992]
  We propose AdaMem, an adaptive user-centric memory framework for long-horizon dialogue agents.<n>AdaMem organizes dialogue history into working, episodic, persona, and graph memories.<n>We evaluate AdaMem on the LoCoMo and PERSONAMEM benchmarks for long-horizon reasoning and user modeling.
  arXiv  Detail & Related papers  (2026-03-17T13:22:54Z)
• From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents [78.30630000529133]
  We propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory.<n> MM-Mem memory structures hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic.<n>Experiments confirm the effectiveness of MM-Mem on both offline and streaming tasks.
  arXiv  Detail & Related papers  (2026-03-02T05:12:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.