StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation

• URL: http://arxiv.org/abs/2609.15509v1
• Date: Mon, 14 Sep 2026 13:01:25 GMT
• Title: StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation
• Abstract summary: We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction.<n>On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding.<n>Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines.
• Score: 4.871965587776022
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in robot imitation learning have produced visuomotor policies that predict actions directly from visual observations. Yet visually similar scenes can require different actions as target position, object height, or contact geometry changes. Pretrained RGB features may map these geometrically distinct states to similar policy inputs, while simply adding depth requires the policy to learn RGB-depth correspondence from the same limited demonstrations used to learn control. We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction. On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding. The resulting StereoPatch Tokens provide a geometry-aware visual representation that can condition general visuomotor policies without changing their underlying learning objectives. Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines. Additional experiments across three simulation suites evaluate compatibility across visuomotor policy architectures, spatial generalization, and operating limits. Results suggest that resolving control-relevant geometric ambiguity benefits from aligning depth directly with the visual features used for action prediction, rather than supplying it as an independent modality. Project page: https://aus.bot/research/stereopatch/.

Related papers

• RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots [11.977612002830725]
  Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as LiDAR or RGB-D cameras, for metric 3D reconstruction.<n>This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only.
  arXiv  Detail & Related papers  (2026-05-18T10:37:39Z)
• StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception [30.791524912385924]
  StereoPolicy is a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to strengthen geometric reasoning.<n>Our results underscore stereo vision as a scalable and robust modality that bridges 2D pretrained representations with 3D geometric understanding for robotic manipulation.
  arXiv  Detail & Related papers  (2026-05-11T05:06:12Z)
• Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy [66.60668908340429]
  Visuomotor policies learn from demonstrations often overfit to nuisance visual factors in raw RGB observations.<n>We propose a task-aware observation interface that canonicalizes visual input into a shared representation.<n>We evaluate on RoboMimic (Lift), ManiSkill YCB grasping under clutter, four RLBench tasks under controlled appearance shifts, and two real-world Franka tasks.
  arXiv  Detail & Related papers  (2026-03-09T01:21:26Z)
• Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation [53.09168514034483]
  Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions.<n>We propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model.<n>Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap.
  arXiv  Detail & Related papers  (2026-02-27T08:54:20Z)
• Learning Category-level Last-meter Navigation from RGB Demonstrations of a Single-instance [5.161531917413708]
  Most RGB-based navigation systems only guarantee coarse, meter-level accuracy.<n>This gap prevents manipulation policies from operating within the distribution of their training demonstrations.<n>We introduce an object-centric imitation learning framework for last-meter navigation.
  arXiv  Detail & Related papers  (2025-12-11T23:35:05Z)
• GLaD: Geometric Latent Distillation for Vision-Language-Action Models [106.53332923530245]
  We introduce GLaD, a geometry-aware Vision-Language-Action (VLA) framework that incorporates 3D geometric priors during pretraining through knowledge distillation.<n>GLaD achieves 94.1% average success rate across four LIBERO task suites, outperforming UniVLA (92.5%) which uses identical pretraining data.
  arXiv  Detail & Related papers  (2025-12-10T13:07:27Z)
• DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation [66.7989548848166]
  Existing approaches encode depth maps along with RGB images and perform feature fusion between them to enable more robust predictions.<n>We present DFormerv2, a strong RGBD encoder that explicitly uses depth maps as geometry priors rather than encoding depth information with neural networks.<n>Our goal is to extract the geometry clues from the depth and spatial distances among all the image patch tokens, which will then be used as geometry priors to allocate attention weights in self-attention.
  arXiv  Detail & Related papers  (2025-04-07T03:06:07Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.