MASt3R-Nav: WayPixel Navigation in Relative 3D Maps

• URL: http://arxiv.org/abs/2605.24111v1
• Date: Fri, 22 May 2026 18:18:07 GMT
• Title: MASt3R-Nav: WayPixel Navigation in Relative 3D Maps
• Abstract summary: We propose a novel map representation in the form of pixel-relative connectivity.<n>Inspired by recent progress in 3D grounded image matching, we construct a map from an image sequence.<n>We show that this dense pixel-level costmap based on relative geometry is a more accurate conditioning variable for control prediction than its image- and object-level counterparts.
• Score: 21.489070961711548
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual navigation ability is strongly tied to its underlying representation of the world. Unlike classical 3D maps that require globally-consistent geometry, image- or object-relative topological graphs almost entirely do away with geometric understanding. But, this comes at the cost of navigation capability, often limiting it to merely teach-and-repeat. In this work, we propose a novel map representation in the form of pixel-relative connectivity, which is geometrically accurate but does not require global geometric consistency. Inspired by recent progress in 3D grounded image matching, we construct a map from an image sequence through inter-image connectivity based on pixel correspondences in the relative 3D coordinate systems of individual image pairs. We then use this pixel-level graph to perform global path planning by approximating and sparsifying intra-image pixel connectivity. Through this, we derive a ''WayPixel Costmap'' representation and train a controller conditioned on it to predict a trajectory rollout. We show that this dense pixel-level costmap based on relative geometry is a more accurate conditioning variable for control prediction than its image- and object-level counterparts. This enables a highly capable navigation system, as validated on four types of navigation tasks in the simulator and through real world demonstrations.

Related papers

• Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence [28.712695548782037]
  We present Tango3D, a foundation model that unifies dense correspondence and global retrieval.<n>We use a geometry-aware 2D visual backbone and a pretrained 3D VAE to encode images into 2D patches and point clouds into 3D tokens.<n>Experiments show our model successfully achieves object-level pixel-to-point alignment while maintaining competitive global retrieval.
  arXiv  Detail & Related papers  (2026-05-19T12:01:23Z)
• IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation [76.36174247570716]
  We propose IVGT, an Implicit Visual Geometry Transformer that implicitly models continuous and coherent geometry from pose-free multi-view images.<n> IVGT learns a continuous neural scene representation in a canonical coordinate system and supports continuous spatial queries at any 3D positions.<n>It allows direct extraction of continuous and coherent surface geometry, enabling rendering of RGB images, depth maps, and surface normal maps from arbitrary viewpoints.
  arXiv  Detail & Related papers  (2026-05-15T17:59:57Z)
• Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels [67.36972154532761]
  Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos.<n>Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking.<n>We propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system.
  arXiv  Detail & Related papers  (2026-03-03T03:45:43Z)
• ObjectReact: Learning Object-Relative Control for Visual Navigation [10.645109937081681]
  We present a new paradigm of learning "object-relative" control that exhibits several desirable characteristics.<n>We propose a topometric map representation in the form of a "relative" 3D scene graph.<n>We demonstrate the advantages of learning object-relative control over its image-relative counterpart across sensor height variations.
  arXiv  Detail & Related papers  (2025-09-11T16:34:17Z)
• IGL-Nav: Incremental 3D Gaussian Localization for Image-goal Navigation [78.00035681410348]
  IGL-Nav is an Incremental 3D Gaussian framework for efficient and 3D-aware image-goal navigation.<n>It can handle the more challenging free-view image-goal setting and be deployed on real-world robotic platform.
  arXiv  Detail & Related papers  (2025-08-01T17:59:56Z)
• Continuous 3D Perception Model with Persistent State [111.83854602049222]
  We present a unified framework capable of solving a broad range of 3D tasks.<n>Our approach features a stateful recurrent model that continuously updates its state representation with each new observation.<n>We evaluate our method on various 3D/4D tasks and demonstrate competitive or state-of-the-art performance in each.
  arXiv  Detail & Related papers  (2025-01-21T18:59:23Z)
• GaussNav: Gaussian Splatting for Visual Navigation [92.13664084464514]
  Instance ImageGoal Navigation (IIN) requires an agent to locate a specific object depicted in a goal image within an unexplored environment.<n>We propose a new framework for IIN, Gaussian Splatting for Visual Navigation (GaussNav), which constructs a novel map representation based on 3D Gaussian Splatting (3DGS)<n>Our GaussNav framework demonstrates a significant performance improvement, with Success weighted by Path Length (SPL) increasing from 0.347 to 0.578 on the challenging Habitat-Matterport 3D (HM3D) dataset.
  arXiv  Detail & Related papers  (2024-03-18T09:56:48Z)
• Monocular Road Planar Parallax Estimation [25.36368935789501]
  Estimating the 3D structure of the drivable surface and surrounding environment is a crucial task for assisted and autonomous driving. We propose Road Planar Parallax Attention Network (RPANet), a new deep neural network for 3D sensing from monocular image sequences. RPANet takes a pair of images aligned by the homography of the road plane as input and outputs a $gamma$ map for 3D reconstruction.
  arXiv  Detail & Related papers  (2021-11-22T10:03:41Z)
• Joint Deep Multi-Graph Matching and 3D Geometry Learning from Inhomogeneous 2D Image Collections [57.60094385551773]
  We propose a trainable framework for learning a deformable 3D geometry model from inhomogeneous image collections. We in addition obtain the underlying 3D geometry of the objects depicted in the 2D images.
  arXiv  Detail & Related papers  (2021-03-31T17:25:36Z)
• Crowdsourced 3D Mapping: A Combined Multi-View Geometry and Self-Supervised Learning Approach [10.610403488989428]
  We propose a framework that estimates the 3D positions of semantically meaningful landmarks without assuming known camera intrinsics. We utilize multi-view geometry as well as deep learning based self-calibration, depth, and ego-motion estimation for traffic sign positioning. We achieve an average single-journey relative and absolute positioning accuracy of 39cm and 1.26m respectively.
  arXiv  Detail & Related papers  (2020-07-25T12:10:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.