Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams

• URL: http://arxiv.org/abs/2606.15200v1
• Date: Sat, 13 Jun 2026 08:50:49 GMT
• Title: Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams
• Abstract summary: UCS-Bench is a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions.<n>We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations.
• Score: 58.77207336324662
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users' real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user's movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.

Related papers

• Dynamic Hub-and-Spoke Memory for Streaming Video Understanding [36.38729564564716]
  Dynamic Hub-and-Spoke Memory (D-HSM) is a training-free framework that represents distant history as structured textual memory.<n>D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory.<n>When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction.
  arXiv  Detail & Related papers  (2026-08-31T06:02:28Z)
• S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval [38.608941713774186]
  We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses.<n>S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream.
  arXiv  Detail & Related papers  (2026-07-02T18:29:20Z)
• EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision [9.701124246177661]
  Continuous episodic memory is a core capability for autonomous agents.<n>Egostream is a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision.
  arXiv  Detail & Related papers  (2026-05-29T17:20:10Z)
• AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents [49.63422082885992]
  We propose AdaMem, an adaptive user-centric memory framework for long-horizon dialogue agents.<n>AdaMem organizes dialogue history into working, episodic, persona, and graph memories.<n>We evaluate AdaMem on the LoCoMo and PERSONAMEM benchmarks for long-horizon reasoning and user modeling.
  arXiv  Detail & Related papers  (2026-03-17T13:22:54Z)
• Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training [61.6942259866261]
  We propose Spatial-TTT towards streaming visual-based spatial intelligence with test-time training (TTT)<n>We design a hybrid architecture and adopt large-temporal updates parallel with sliding-window attention for efficient spatial video processing.<n>Experiments demonstrate that Spatial-TTT improves long-horizon spatial understanding and achieves state-of-the-art performance on video spatial benchmarks.
  arXiv  Detail & Related papers  (2026-03-12T17:58:58Z)
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT [56.24624833924252]
  EgoThinker is a framework that endows MLs with robust egocentric reasoning capabilities through-temporal chain-of-thought supervision and a two-stage learning curriculum.<n>EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained-temporal localization tasks.
  arXiv  Detail & Related papers  (2025-10-27T17:38:17Z)
• Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data [100.5266292850922]
  Strefer is a synthetic data generation framework designed to equip Video Large Models with referring and reasoning capabilities.<n>Strefer produces diverse instruction-generation data using a data engine that pseudo-annotates temporally dense, fine-grained video metadata.<n>Our approach enhances the ability of Video LLMs to interpret to spatial and temporal references, fostering more versatile, space-time-aware reasoning essential for real-world AI companions.
  arXiv  Detail & Related papers  (2025-09-03T17:33:20Z)
• Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos [51.8995932557911]
  EgoTempo is a dataset designed to evaluate temporal understanding in the egocentric domain.<n>We show that state-of-the-art Multi-Modal Large Language Models (MLLMs) on benchmarks achieve remarkably high performance using just text or a single frame as input.<n>We hope EgoTempo will catalyze new research in the field and inspire models that better capture the complexity of temporal dynamics.
  arXiv  Detail & Related papers  (2025-03-17T18:50:36Z)
• EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset [19.496721051685135]
  Embodied tracking is a key component to many egocentric vision problems. EgoTracks is a new dataset for long-term egocentric visual object tracking. We show improvements that can be made to a STARK tracker to significantly increase its performance on egocentric data.
  arXiv  Detail & Related papers  (2023-01-09T09:10:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.