Visual Navigation Transformer with Pose Attention
Abstract Overview
This paper proposes VNT-PA, a visual navigation transformer that represents an environment as a set of depth keyframes indexed by camera pose rather than as a temporally ordered observation history or an explicit map. The model uses camera poses as rotary positional embeddings, so attention depends on pose differences between frames, and it predicts actions by querying this spatial context with only the robot pose and goal position. It is trained to imitate a shortest-path planner operating on ground-truth scene meshes and evaluated on point-goal navigation in HM3D validation scenes. The study focuses on whether pose-stamped prior observations can directly support planning in previously explored environments.
Novelty
The distinctive contribution is treating pose-stamped keyframes themselves as the environment representation for a learned planner, without converting them into an explicit map or organizing them as a temporal sequence. The paper also makes pose differences, rather than time order or pose-as-feature inputs, the basis of transformer attention through pose-based rotary positional embeddings.
Results
On HM3D validation scenes, VNT-PA achieves 93.3% success and 90.4% SPL, outperforming baselines that use the same context as a temporal sequence, a recurrent state, or pose as an input feature. It also reaches strong performance faster during training, benefits from adding extra frames at test time, and degrades less under localization noise than a conventional OctoMap plus A* pipeline built from the same observations.
Key Points
- VNT-PA organizes prior depth observations as a pose-indexed spatial context and queries it directly for navigation decisions, without requiring an explicit map or current visual input.
- Using camera pose as rotary positional embedding provides a useful inductive bias, improving long-horizon and detour-heavy navigation relative to temporal attention and pose-as-feature baselines.
- The representation is flexible: additional frames from earlier trajectories or online robot views can be fused at test time, and the method remains comparatively robust under localization error.