Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation

• URL: http://arxiv.org/abs/2608.02257v1
• Date: Mon, 03 Aug 2026 14:03:26 GMT
• Title: Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation
• Abstract summary: Mobile manipulation poses two key challenges for vision-language-action policies.<n>We develop a whole-body teleoperation system and a panoramic-aware VLA policy.<n>We propose PanoVLA, a panorama-aware vision-language-action policy for mobile bimanual manipulation.
• Score: 55.724481205135675
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mobile manipulation is a key capability for embodied intelligence, enabling robots to accomplish complex multi-stage tasks in open-world environments. However, mobile manipulation poses two key challenges for vision-language-action (VLA) policies: At the data level, the efficient collection of high-quality whole-body demonstrations demands the coordinated control of both the mobile base and the robotic arms; at the model level, existing VLA models predominantly rely on local camera observations, whose limited field of view hinders global spatial understanding. To address these challenges, we develop a whole-body teleoperation system and a panoramic-aware VLA policy. The system enables coordinated control of a wheeled bimanual robot through a single VR interface and supports the acquisition of a real-world mobile manipulation dataset comprising 5.5 hours of multimodal demonstrations. Building upon this dataset, we propose PanoVLA, a panorama-aware vision-language-action policy for mobile bimanual manipulation. Built upon a Mixture-of-Transformers architecture, PanoVLA introduces global spatial context through dedicated panorama encoding and fusion modules, enabling effective integration of panoramic observations with language instructions and robot states for action generation. Evaluation on four real-world mobile manipulation tasks demonstrates that PanoVLA achieves an average stage completion rate of 91.3\% and an end-to-end success rate of 73.4\%, substantially outperforming local-view baselines. These results demonstrate that incorporating panoramic spatial context improves spatial understanding and closed-loop manipulation performance in mobile robots.

Related papers

• GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations [19.347332945780273]
  Vision-Language-Action (VLA) models have shown strong potential for general-purpose robot manipulation by unifying perception and action.<n>We introduce gesture as a parallel instruction modality and propose a Gesture-aware Vision-Language-Action model (GesVLA)<n> Experimental results show that incorporating gesture consistently improves target grounding accuracy and human-robot interaction efficiency.
  arXiv  Detail & Related papers  (2026-05-21T17:57:44Z)
• SG-VLA: Learning Spatially-Grounded Vision-Language-Action Models for Mobile Manipulation [43.77409401429281]
  Vision-Language-Action (VLA) models show promise for robotic control, yet performance in complex household environments remains sub-optimal.<n>Mobile manipulation requires reasoning about global scene layout, fine-grained geometry, and high-dimensional continuous actions.<n>We introduce a framework for learning spatially-grounded VLA models that strengthens perception and representation.
  arXiv  Detail & Related papers  (2026-03-24T03:44:25Z)
• RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation [104.68774434699158]
  RoboInter Manipulation Suite is a unified resource including data, benchmarks, and models of intermediate representations for manipulation.<n>It comprises RoboInter-Tool, a lightweight GUI that enables semi-automatic annotation of diverse representations, and RoboInter-Data, a large-scale dataset containing over 230k episodes across 571 diverse scenes.<n>RoboInter-VLA offers an integrated plan-then-execute framework, supporting modular and end-to-end VLA variants.
  arXiv  Detail & Related papers  (2026-02-10T17:01:54Z)
• MobileManiBench: Simplifying Model Verification for Mobile Manipulation [70.30578259859512]
  MobileManiBench is a large-scale benchmark for mobile-based robotic manipulation.<n>MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories.
  arXiv  Detail & Related papers  (2026-02-05T02:49:52Z)
• HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies [83.41714103649751]
  Development of embodied intelligence models depends on access to high-quality robot demonstration data.<n>We present HiMoE-VLA, a novel vision-language-action framework tailored to handle diverse robotic data with heterogeneity.<n>HiMoE-VLA demonstrates a consistent performance boost over existing VLA baselines, achieving higher accuracy and robust generalizations.
  arXiv  Detail & Related papers  (2025-12-05T13:21:05Z)
• End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collection [10.217810309422232]
  We propose a framework that divides control between macro and micro motions.<n>A human operator guides the robot's arm pose through intuitive VR teleoperation.<n>An autonomous DexGrasp-VLA policy handles fine-grained hand control using real-time tactile and visual feedback.
  arXiv  Detail & Related papers  (2025-10-31T16:12:02Z)
• M4Diffuser: Multi-View Diffusion Policy with Manipulability-Aware Control for Robust Mobile Manipulation [17.9979990426915]
  M4Diffuser is a hybrid framework that integrates a Multi-View Diffusion Policy with a novel Reduced and Manipulability-aware QP controller for mobile manipulation.<n>Our approach demonstrates robust performance for smooth whole-body coordination, and strong generalization to unseen tasks.
  arXiv  Detail & Related papers  (2025-09-18T14:09:53Z)
• OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis [70.39500621448383]
  Open-world mobile manipulation task remains a challenge due to the need for generalization to open-ended instructions and environments.<n>We propose a novel multi-modal agent architecture that maintains multi-view scene frames and agent states for decision-making and controls the robot by function calling.<n>We highlight our fine-tuned OWMM-VLM as the first dedicated foundation model for mobile manipulators with global scene understanding, robot state tracking, and multi-modal action generation in a unified model.
  arXiv  Detail & Related papers  (2025-06-04T17:57:44Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.