Stereo Multistage Spatial Attention for Real-Time Mobile Manipulation Under Visual Scale Variation and Disturbances

• URL: http://arxiv.org/abs/2605.00471v1
• Date: Fri, 01 May 2026 07:18:11 GMT
• Title: Stereo Multistage Spatial Attention for Real-Time Mobile Manipulation Under Visual Scale Variation and Disturbances
• Abstract summary: We present a stereo multistage spatial attention-based deep predictive learning method for real-time mobile manipulation.<n>The proposed methods extracts task-relevant spatial attention points from stereo images and integrates them with robot states.<n>We evaluate the system on four real-world mobile manipulation tasks using a mobile manipulator.
• Score: 5.013648961622839
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robots operating in open, unstructured real-world environments must rely on onboard visual perception while autonomously moving across different locations. Continuous changes in onboard camera viewpoints cause significant visual scale variations in target objects, affecting vision-based motion generation. In this work, we present a stereo multistage spatial attention-based deep predictive learning method for real-time mobile manipulation. The proposed methods extracts task-relevant spatial attention points from stereo images and integrates them with robot states through a hierarchical recurrent architecture for closed-loop action prediction. We evaluate the system on four real-world mobile manipulation tasks using a mobile manipulator, including rigid placement, articulated object manipulation, and deformable object interaction. Experiments under randomized initial positions and visual disturbance conditions demonstrate improved robustness and task success rates compared to representative imitation learning and vision-language-action baselines under identical control settings. The results indicate that structured stereo spatial attention combined with predictive temporal modeling provides an effective solution within the evaluated mobile manipulation scenarios.

Related papers

• Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation [55.724481205135675]
  Mobile manipulation poses two key challenges for vision-language-action policies.<n>We develop a whole-body teleoperation system and a panoramic-aware VLA policy.<n>We propose PanoVLA, a panorama-aware vision-language-action policy for mobile bimanual manipulation.
  arXiv  Detail & Related papers  (2026-08-03T14:03:26Z)
• Masked Visual Actions for Unified World Modeling [96.12988421978463]
  We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video.<n>Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome.
  arXiv  Detail & Related papers  (2026-07-21T17:59:11Z)
• ABot-M0.5: Unified Mobility-and-Manipulation World Action Model [27.262271208923995]
  ABot-M0.5 is built on the insight that mobile manipulation requires alignment at three levels: temporal granularity, action space, and train-test consistency.<n> Experiments on challenging mobile and fine-grained manipulation benchmarks demonstrate that ABot-M0.5 achieves state-of-the-art performance in both long-horizon task success and finegrained control accuracy.
  arXiv  Detail & Related papers  (2026-07-01T09:21:20Z)
• Joint Discovery of Object and Action Symbols through Effect Prediction for Robotic Manipulation Planning [1.4701783773624093]
  We propose a model that jointly discovers high-level manipulation primitives and object categories.<n>We leverage a discrete planning method that uses intermediate steps in the predicted effect trajectory to enable partial action executions.<n>We conduct experiments on tabletop repositioning and stacking tasks, and confirm that our effect-driven planning approach outperforms both a state-of-the-art method and a visual-based alternative.
  arXiv  Detail & Related papers  (2026-06-22T11:58:11Z)
• ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration [51.69384671495837]
  ActiveGlasses is a system for learning robot manipulation from ego-centric human demonstrations with active vision.<n>A stereo camera mounted on smart glasses serves as the sole perception device for both data collection and policy inference.<n>To enable zero-transfer, we extract object trajectories from demonstrations and use an object-centric point-cloud policy to jointly predict manipulation and head movement.
  arXiv  Detail & Related papers  (2026-04-09T17:59:08Z)
• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation [57.28703268044067]
  We propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation.<n>Our framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
  arXiv  Detail & Related papers  (2026-03-22T04:18:54Z)
• I-Perceive: A Foundation Model for Active Perception with Language Instructions [41.67607728608853]
  We propose I-Perceive, a foundation model for active perception conditioned on natural language instructions.<n>I-Perceive predicts camera views that follow open-ended language instructions, based on image-based scene contexts.<n> Experiments demonstrate that I-Perceive significantly outperforms state-of-the-art VLMs in both prediction accuracy and instruction following of generated camera views.
  arXiv  Detail & Related papers  (2026-02-28T11:38:56Z)
• Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress [53.723881111373736]
  We present SPARTA, the first unified framework for the family of object state change manipulation tasks.<n>SPARTA integrates spatially progressing object change segmentation maps, a visual skill to perceive actionable vs. transformed regions, and dense rewards that capture incremental progress over time.<n>We validate SPARTA on a real robot for three challenging tasks across 10 diverse real-world objects.
  arXiv  Detail & Related papers  (2025-09-28T23:56:07Z)
• Towards an Accurate and Effective Robot Vision (The Problem of Topological Localization for Mobile Robots) [0.43064121494080315]
  This work addresses topological localization in an office environment using only images acquired with a perspective color camera mounted on a robot platform.<n>We evaluate state-of-the-art visual descriptors, including Color Histograms, SIFT, ASIFT, RGB-SIFT, and Bag-of-Visual-Words approaches inspired by text retrieval.
  arXiv  Detail & Related papers  (2025-09-05T09:14:59Z)
• AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation [31.314066269767057]
  Mobile manipulation has attracted increasing attention for enabling language-conditioned robotic control in household tasks.<n>Existing methods fail to explicitly model the influence of the mobile base on manipulator control.<n>We propose the Adaptive Coordination Diffusion Transformer (AC-DiT) to enhance mobile base and manipulator coordination.
  arXiv  Detail & Related papers  (2025-07-02T17:59:54Z)
• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [73.75271615101754]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences.<n>Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations.<n>Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• Diffusion Transformer Policy [48.50988753948537]
  We propose a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, to model continuous end-effector actions.<n>By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets.
  arXiv  Detail & Related papers  (2024-10-21T12:43:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.