RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics

• URL: http://arxiv.org/abs/2603.12939v1
• Date: Fri, 13 Mar 2026 12:34:26 GMT
• Title: RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
• Abstract summary: VLM-based planners treat each step as an observation-to-action mapping, forcing them to reinfer scene geometry from raw pixels at every decision point.<n>We propose SStream-Temporal Fusion Tokens (TF-Tokens), which bind visual evidence to 3D geometric attributes for persistent object grounding.<n>This design enables the planner to trace causal chains and preserve object permanence under occlusion without additional training or fine-tuning.
• Score: 25.944338247511215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enabling reliable long-horizon robotic manipulation is a crucial step toward open-world embodied intelligence. However, VLM-based planners treat each step as an isolated observation-to-action mapping, forcing them to reinfer scene geometry from raw pixels at every decision point while remaining unaware of how prior actions have reshaped the environment. Despite strong short-horizon performance, these systems lack the spatio-temporal reasoning required for persistent geometric anchoring and memory of action-triggered state transitions. Without persistent state tracking, perceptual errors accumulate across the execution horizon, temporarily occluded objects are catastrophically forgotten, and these compounding failures lead to precondition violations that cascade through subsequent steps. In contrast, humans maintain a persistent mental model that continuously tracks spatial relations and action consequences across interactions rather than reconstructing them at each instant. Inspired by this human capacity for causal spatio-temporal reasoning with persistent memory, we propose RoboStream, a training-free framework that achieves geometric anchoring through Spatio-Temporal Fusion Tokens (STF-Tokens), which bind visual evidence to 3D geometric attributes for persistent object grounding, and maintains causal continuity via a Causal Spatio-Temporal Graph (CSTG) that records action-triggered state transitions across steps. This design enables the planner to trace causal chains and preserve object permanence under occlusion without additional training or fine-tuning. RoboStream achieves 90.5% on long-horizon RLBench and 44.4% on challenging real-world block-building tasks, where both SoFar and VoxPoser score 11.1%, demonstrating that spatio-temporal reasoning and causal memory are critical missing components for reliable long-horizon manipulation.

Related papers

• Human-Aware Target Tracking and Navigation: Fusing Kinematic State Estimation with Structural Map Constraints [3.9308813646788625]
  This research presents an end-to-end autonomous navigation stack that addresses target occlusion through map-informed spatial reasoning.<n>The proposed system features a multi-modal perception pipeline, fusing deep learning-based visual tracking with 2-dimensional LiDAR point clustering to maintain high-fidelity tracking of a tagged person.<n>Real-world evaluations in dense multi-person environments demonstrate the system's robustness, achieving a 71.4% target reacquisition success rate.
  arXiv  Detail & Related papers  (2026-09-07T06:31:17Z)
• LT-Mem: Volatility-Aware Spatio-Temporal Memory for Lifelong Scene Understanding [3.1007425074240866]
  Long-term robot operation in evolving environments requires object-level understanding that persists across repeated revisits.<n>Existing systems either overwrite history to maintain an up-to-date map or store semantic snapshots without consistent cross-session object identity.<n>We propose LT-Mem, a volatility-aware memory evolution framework that unifies spatially aligned instance-level 3D perception with volatility-conditioned temporal reasoning.
  arXiv  Detail & Related papers  (2026-08-19T15:56:53Z)
• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy [79.09714980186278]
  Building persistent embodied agents in unstructured environments requires unified orchestration of heterogeneous tools.<n>We present OmniAct, a framework integrating a multimodal semantic planner for skill routing across unified action spaces.<n>It achieves consistent improvements in end-to-end success across all complexity levels.
  arXiv  Detail & Related papers  (2026-06-25T16:36:35Z)
• SCOUT: Semantic scene COverage via Uncertainty-guided Traversal [2.499955687541552]
  Most 3D scene graph pipelines treat perception as a post-processing stage over a fixed dataset.<n>We present SCOUT, an online semantic exploration framework that closes this loop.<n>We show that SCOUT treats semantic scene completeness as an operational objective rather than a passive by-product of semantic mapping.
  arXiv  Detail & Related papers  (2026-06-04T21:13:33Z)
• Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation [89.70897512515477]
  Action-controlled image-to-video generation is a promising paradigm for interactive world simulation, where each control signal should elicit an immediate visual response.<n>Maintaining visual fidelity and 3D consistency over long autoregressive rollouts remains challenging.<n>Existing 3D-aware methods often suffer from catastrophic drift due to two impediments: information loss from textitLatent--RGB Cycling, and the training--inference gap induced by the textiterror-free hypothesis.<n>We present textbfRobust Dreamer, a memory-augmented framework built around how to
  arXiv  Detail & Related papers  (2026-05-29T05:21:33Z)
• Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors [51.096845970243855]
  Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data.<n>We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction.
  arXiv  Detail & Related papers  (2026-05-21T10:15:39Z)
• ANCHOR: A Physically Grounded Closed-Loop Framework for Robust Home-Service Mobile Manipulation [10.278758849939917]
  We present ANCHOR, a closed-loop framework that aligns symbolic reasoning with verifiable physical state during execution.<n>ANCHOR improves task success from 53.3% to 71.7% and achieves a 71.4% recovery rate under perturbations.
  arXiv  Detail & Related papers  (2026-04-28T07:37:39Z)
• End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction [74.31251139839047]
  THO is an end-to-end Spatial-Temporal Transformer that predicts human motion and coordinated object motion in a forward fashion from the given video and 3D template.<n> experiments demonstrate that THO operates at an inference speed of 31.5 FPS on a single 4090 GPU, achieving a >600x speedup over prior optimization-based methods.
  arXiv  Detail & Related papers  (2026-03-15T15:21:36Z)
• OccSTeP: Benchmarking 4D Occupancy Spatio-Temporal Persistence [19.81329748424203]
  We introduce a new concept of 4D Occupancy Spatio-Temporal Persistence (OccSTeP)<n>This concept aims to address two tasks: (1) reactive forecasting: ''what happen'' and (2) proactive forecasting: "what would happen given a specific future action"<n>We propose OccSTeP-WM, a tokenizer-free world model that maintains a dense voxel-based scene state and fuses incrementally-temporal context over time.
  arXiv  Detail & Related papers  (2025-12-17T17:29:20Z)
• ST-DETrack: Identity-Preserving Branch Tracking in Entangled Plant Canopies via Dual Spatiotemporal Evidence [0.7448758790630549]
  We propose ST-DETrack, a dual-decoder network designed to preserve branch identity from budding to flowering.<n>Our architecture integrates spatial consistency and a temporal decoder that exploits motion ambiguities.<n> ST-DETrack achieves a Branch Matching Accuracy (BMA) of 93.6%, significantly outperforming spatial and temporal baselines by 28.9 and 3.3 percentage points, respectively.
  arXiv  Detail & Related papers  (2025-12-17T13:42:34Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective [16.541717037293278]
  We introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability.<n>It combines short- and long-horizon object tracking with temporally sequenced subgoals, requiring reasoning beyond the current frame.<n>We propose Embodied-SlotSSM, a slot-centric VLA framework built for temporal scalability.
  arXiv  Detail & Related papers  (2025-11-14T16:56:01Z)
• Long-Context State-Space Video World Models [66.28743632951218]
  We propose a novel architecture leveraging state-space models (SSMs) to extend temporal memory without compromising computational efficiency.<n>Central to our design is a block-wise SSM scanning scheme, which strategically trades off spatial consistency for extended temporal memory.<n>Experiments on Memory Maze and Minecraft datasets demonstrate that our approach surpasses baselines in preserving long-range memory.
  arXiv  Detail & Related papers  (2025-05-26T16:12:41Z)
• Spatio-Temporal Bi-directional Cross-frame Memory for Distractor Filtering Point Cloud Single Object Tracking [2.487142846438629]
  3 single object tracking within LIDAR point is pivotal task in computer vision. Existing methods, which depend solely on appearance matching via networks or utilize information from successive frames, encounter significant challenges. We design an innovative cross-frame bi-temporal motion tracker, named STMD-Tracker, to mitigate these challenges.
  arXiv  Detail & Related papers  (2024-03-23T13:15:44Z)
• Nothing Stands Still: A Spatiotemporal Benchmark on 3D Point Cloud Registration Under Large Geometric and Temporal Change [82.31647863785923]
  Building 3D geometric maps of man-made spaces are fundamental computer vision and robotics.<n>Nothing Stands Still (NSS) benchmark focuses on thetemporal registration of 3D scenes undergoing large spatial and temporal change.<n>As part of NSS, we introduce a dataset of 3D point clouds recurrently captured in large-scale building indoor environments that are under construction or renovation.
  arXiv  Detail & Related papers  (2023-11-15T20:09:29Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.