Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents

• URL: http://arxiv.org/abs/2609.04802v1
• Date: Fri, 04 Sep 2026 07:00:48 GMT
• Title: Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents
• Abstract summary: Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language.<n>Our key contribution is textbfLinguistic Trajectory Linguistic Trajectory (LTE)<n>LTE compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors.<n>LTE achieves trajectory compression by factors of $8.7times to $26.1times with sub-second query latency on $24,h video.
• Score: 24.05637675858576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\%$ success in semantic trajectory retrieval and $48.7\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\%$ and $34.4\%$). LTE achieves trajectory compression by factors of $8.7\times$ to $26.1\times$ with sub-second query latency on $24$\,h video. On Ego4D natural-language queries, the system reaches $28.75\%$ / $55.10\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.

Related papers

• RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memory [10.92726806758712]
  RAVEN is an agentic memory system for long-horizon robotic question answering and navigation.<n>RAVEN stores visual embeddings with pose and time in a vector database, and grounds retrieval in a spatial map to answer queries.<n>We instantiate RAVEN on a Unitree Go1 robot for the task of long-horizon navigation for natural language goal-reaching.
  arXiv  Detail & Related papers  (2026-06-23T21:56:41Z)
• Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams [58.77207336324662]
  UCS-Bench is a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions.<n>We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations.
  arXiv  Detail & Related papers  (2026-06-13T08:50:49Z)
• GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0) [56.81743709880371]
  Long-horizon large language model (LLM) agents are fundamentally limited by context.<n>We argue that long-horizon performance is determined not by context length, but by how much decision-relevant information is maintained within a finite context budget.<n>We present GenericAgent (GA), a general-purpose, self-evolving LLM agent system built around a single principle: information density and context.
  arXiv  Detail & Related papers  (2026-04-18T17:59:15Z)
• Semantic Event Graphs for Long-Form Video Question Answering [0.0]
  We propose a lightweight symbolic interface between video and language that replaces raw frames with compact temporal interaction logs.<n>Our pipeline detects and tracks objects with YOLOv11, converts proximity patterns into START/END human-object events, and organizes them into a Temporal Scene Graph (TSG)<n>On five YouTube videos (300-500 interactions each) and 120 automatically generated long-horizon questions, SEG 65.0% accuracy using only 3.47k tokens per query, closely matching a full-log baseline and reducing token usage by 91.4%.
  arXiv  Detail & Related papers  (2026-01-02T00:11:03Z)
• LAVA: Language Driven Scalable and Versatile Traffic Video Analytics [43.754808270319664]
  We build textscLava, a system that accepts natural language queries and retrieves traffic targets.<n>textscLava comprises three main components: 1) a multi-armed bandit-based efficient sampling method for video segment-level localization; 2) a video-specific open-world detection module for object-level retrieval; and 3) a long-term object trajectory extraction scheme for temporal object association.
  arXiv  Detail & Related papers  (2025-07-26T06:38:07Z)
• Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation [51.2732688481343]
  We introduce Longtextbf-RVOS, a large-scale benchmark for long-term referring object segmentation.<n>Long-RVOS contains 2,000+ videos of an average duration exceeding 60 seconds, covering a variety of objects.<n>Unlike previous benchmarks that rely solely on the per-frame spatial evaluation, we introduce two metrics to assess the temporal andtemporal consistency.
  arXiv  Detail & Related papers  (2025-05-19T04:52:31Z)
• Temporally-Grounded Language Generation: A Benchmark for Real-Time Vision-Language Models [14.69446469340837]
  Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering.<n>Real-time interactive environments impose new demands on VLMs, requiring them to generate utterances that are not only semantically accurate but also precisely timed.<n>We present $textbfVision-Language Model with Time-Synchronized Interleaving (VLM-TSI)$, a model that interleaves visual and linguistic tokens in a time-synchronized manner.
  arXiv  Detail & Related papers  (2025-05-16T14:48:30Z)
• SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability [58.46310813774538]
  Large language models (LMLMs) have made remarkable progress in either temporal or spatial localization.<n>However they struggle to perform-temporal video grounding.<n>This limitation stems from two major challenges.<n>We introduce SpaceLM, a MLLMVL endowed with temporal-temporal video grounding.
  arXiv  Detail & Related papers  (2025-03-18T07:40:36Z)
• Learning Spatial-Semantic Features for Robust Video Object Segmentation [108.045326229865]
  We propose a robust video object segmentation framework that learns spatial-semantic features and discriminative object queries.<n>The proposed method achieves state-of-the-art performance on benchmark data sets, including the DAVIS 2017 test (textbf87.8%), YoutubeVOS 2019 (textbf88.1%), MOSE val (textbf74.0%), and LVOS test (textbf73.0%)
  arXiv  Detail & Related papers  (2024-07-10T15:36:00Z)
• ObjectNLQ @ Ego4D Episodic Memory Challenge 2024 [51.57555556405898]
  We present our approach for the Natural Language Query track and Goal Step track of the Ego4D Episodic Memory Benchmark at CVPR 2024. Both challenges require the localization of actions within long video sequences using textual queries. We introduce a novel approach, termed ObjectNLQ, which incorporates an object branch to augment the video representation with detailed object information.
  arXiv  Detail & Related papers  (2024-06-22T07:57:58Z)
• Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Grounding [56.315932539150324]
  We design a Unified Static and Dynamic Network (UniSDNet) to learn the semantic association between the video and text/audio queries.<n>Our UniSDNet is applicable to both Natural Language Video Grounding (NLVG) and Spoken Language Video Grounding (SLVG) tasks.
  arXiv  Detail & Related papers  (2024-03-21T06:53:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.