SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

• URL: http://arxiv.org/abs/2606.12956v1
• Date: Thu, 11 Jun 2026 06:29:49 GMT
• Title: SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation
• Abstract summary: We show that conditioning a mobile manipulation policy on atemporal feature map improves reasoning over long horizons.<n>We use our environment and robot feature (SERF) map as a state input to a vision-language-action model.<n>We demonstrate SERF on BEHAVIOR-1K, a benchmark for long-horizon mobile manipulation in household environments.
• Score: 11.94873222185677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon robot mobile manipulation requires continual reasoning about localization, environment changes, and task progress, all of which are challenging to infer from image observations alone. In this paper, we show that conditioning a mobile manipulation policy on a spatiotemporal feature map improves reasoning over long horizons. The map represents the environment and the articulated robot body as neural points in a shared latent space and is updated online from egocentric observations and proprioceptive state. We update the environment neural points using object-level rigid tracking and the robot neural points using forward kinematics. We use our spatiotemporal environment and robot feature (SERF) map as a state input to a vision-language-action (VLA) model by extracting map tokens from multiple reference frames and spatial scales, providing the policy with both local and global context. We demonstrate SERF on BEHAVIOR-1K, a benchmark for long-horizon mobile manipulation in household environments. Experiments show that the SERF VLA policy outperforms image-only baselines, reaches subgoals faster by following more direct trajectories, improves robustness to scene-configuration shifts, and recovers from object-drop failures.

Related papers

• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• Temporally-Sampled Efficiently Adaptive State Lattices for Autonomous Ground Robot Navigation in Partially Observed Environments [2.2188568806866287]
  In traditional autonomous navigation architectures, a regional motion planner will consume the environment map and output a trajectory for the local motion planner to use as a reference.<n>We propose Temporally-Sampled Efficiently Adaptive State Lattices (TSEASL) to consider updated and optimized versions of previously generated trajectories against the currently generated trajectory.<n>Results indicate that when running TSEASL, the robot did not require manual interventions in the same locations where the robot was running the baseline planner.
  arXiv  Detail & Related papers  (2026-02-13T18:14:56Z)
• Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments [10.953629652228024]
  Vision-and-Language Navigation (VLN) agents associate time-sequenced visual observations with corresponding instructions to make decisions.<n>In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view.<n>We propose a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue.
  arXiv  Detail & Related papers  (2025-02-26T10:30:40Z)
• Watch Your STEPP: Semantic Traversability Estimation using Pose Projected Features [4.392942391043664]
  We propose a method for estimating terrain traversability by learning from demonstrations of human walking.<n>Our approach leverages dense, pixel-wise feature embeddings generated using the DINOv2 vision Transformer model.<n>By minimizing loss, the network distinguishes between familiar terrain with a low reconstruction error and unfamiliar or hazardous terrain with a higher reconstruction error.
  arXiv  Detail & Related papers  (2025-01-29T11:53:58Z)
• Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model [63.336123527432136]
  We introduce Bench2Drive-R, a generative framework that enables reactive closed-loop evaluation.<n>Unlike existing video generative models for autonomous driving, the proposed designs are tailored for interactive simulation.<n>We compare the generation quality of Bench2Drive-R with existing generative models and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-11T06:35:18Z)
• Outlier-Robust Long-Term Robotic Mapping Leveraging Ground Segmentation [1.7948767405202701]
  I propose a robust long-term robotic mapping system that can work out of the box. I propose (i) fast and robust ground segmentation to reject the presence of outliers. I propose (ii)-robust registration with ground segmentation that encompasses the presence of gross outliers.
  arXiv  Detail & Related papers  (2024-05-18T04:56:15Z)
• NeuPAN: Direct Point Robot Navigation with End-to-End Model-based Learning [67.53972459080437]
  Navigating a nonholonomic robot in a cluttered, unknown environment requires accurate perception and precise motion for real-time collision avoidance.<n>This paper presents NeuPAN: a real-time, highly accurate, map-free, easy-to-deploy, and environment-invariant robot motion planner.
  arXiv  Detail & Related papers  (2024-03-11T15:44:38Z)
• Spatiotemporal-Augmented Graph Neural Networks for Human Mobility Simulation [35.89805766554052]
  We propose a novel framework to model the dynamictemporal effects of locations, namely SRpatio-Augmented gaph neural networks. The STAR framework designs varioustemporal graphs to capture the behaviors correspondence and builds a novel branch to simulate the varying dwells in locations, which duration is finally optimized in an adversarial manner.
  arXiv  Detail & Related papers  (2023-06-15T11:47:45Z)
• Neural Scene Representation for Locomotion on Structured Terrain [56.48607865960868]
  We propose a learning-based method to reconstruct the local terrain for a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the estimates the topography in the robot's vicinity. We propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement.
  arXiv  Detail & Related papers  (2022-06-16T10:45:17Z)
• Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation [87.03299519917019]
  We propose a dual-scale graph transformer (DUET) for joint long-term action planning and fine-grained cross-modal understanding. We build a topological map on-the-fly to enable efficient exploration in global action space. The proposed approach, DUET, significantly outperforms state-of-the-art methods on goal-oriented vision-and-language navigation benchmarks.
  arXiv  Detail & Related papers  (2022-02-23T19:06:53Z)
• Rapid Exploration for Open-World Navigation with Latent Goal Models [78.45339342966196]
  We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration.
  arXiv  Detail & Related papers  (2021-04-12T23:14:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.