AURA: Multimodal Shared Autonomy for Real-World Urban Navigation

• URL: http://arxiv.org/abs/2604.01659v1
• Date: Thu, 02 Apr 2026 05:59:31 GMT
• Title: AURA: Multimodal Shared Autonomy for Real-World Urban Navigation
• Abstract summary: Long-horizon navigation in complex urban environments relies heavily on continuous human operation.<n>We present Assistive Urban Robot Autonomy (AURA), a new multi-modal framework that decomposes urban navigation into high-level human instruction and low-level AI control.<n>AURA effectively follows human instructions, reduces manual operation effort, and improves navigation stability, while enabling online adaptation.
• Score: 49.419374037785815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon navigation in complex urban environments relies heavily on continuous human operation, which leads to fatigue, reduced efficiency, and safety concerns. Shared autonomy, where a Vision-Language AI agent and a human operator collaborate on maneuvering the mobile machine, presents a promising solution to address these issues. However, existing shared autonomy methods often require humans and AI to operate within the same action space, leading to high cognitive overhead. We present Assistive Urban Robot Autonomy (AURA), a new multi-modal framework that decomposes urban navigation into high-level human instruction and low-level AI control. AURA incorporates a Spatial-Aware Instruction Encoder to align various human instructions with visual and spatial context. To facilitate training, we construct MM-CoS, a large-scale dataset comprising teleoperation and vision-language descriptions. Experiments in simulation and the real world demonstrate that AURA effectively follows human instructions, reduces manual operation effort, and improves navigation stability, while enabling online adaptation. Moreover, under similar takeover conditions, our shared autonomy framework reduces the frequency of takeovers by more than 44%. Demo video and more detail are provided in the project page.

Related papers

• TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model [49.03909082056646]
  We study the problem of navigating cluttered indoor environments with a humanoid robot.<n>We introduce TANGO, the first whole-body vision-language navigation framework.<n>In simulation experiments, TANGO demonstrates state-of-the-art performance in vision-language navigation.
  arXiv  Detail & Related papers  (2026-09-08T17:59:55Z)
• Flying Together: Human-Guided Immersive Shared Control for Aerial Robot Teams in Unknown Environments [12.913935671942774]
  We present a Virtual Reality (VR)-based shared control framework for teams of drones operating in constrained and unknown environments.<n>A novel, user-guided motion-primitive-based planner computes continuous, collision-free trajectories while continuously integrating operator input.<n>The system supports mixed-reality operations with both physical and simulated drones, and implements a bilateral VR-based interface.
  arXiv  Detail & Related papers  (2026-05-20T19:37:46Z)
• AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild [62.47761809929869]
  Vision-language navigation (VLN) requires intelligent agents to navigate environments by interpreting linguistic instructions alongside visual observations.<n>Current VLN research for unmanned aerial vehicles (UAVs) relies on detailed, pre-specified instructions to guide the UAV along predetermined routes.<n>We propose AutoFly, an end-to-end Vision-Language-Action model for autonomous UAV navigation.
  arXiv  Detail & Related papers  (2026-02-10T11:08:07Z)
• Follow-Me in Micro-Mobility with End-to-End Imitation Learning [0.0]
  We show how imitation learning delivers smoother and overall better controllers, versus previously used manually-tuned controllers.<n>This paper analyzes different neural network architectures for end-to-end control and demonstrates their usability in real-world production-level deployments.
  arXiv  Detail & Related papers  (2025-11-07T11:24:20Z)
• End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collection [10.217810309422232]
  We propose a framework that divides control between macro and micro motions.<n>A human operator guides the robot's arm pose through intuitive VR teleoperation.<n>An autonomous DexGrasp-VLA policy handles fine-grained hand control using real-time tactile and visual feedback.
  arXiv  Detail & Related papers  (2025-10-31T16:12:02Z)
• Towards Autonomous Micromobility through Scalable Urban Simulation [52.749987132021324]
  Current micromobility depends mostly on human manual operation (in-person or remote control)<n>In this work, we present a scalable urban simulation solution to advance autonomous micromobility.
  arXiv  Detail & Related papers  (2025-05-01T17:52:29Z)
• IG-MCTS: Human-in-the-Loop Cooperative Navigation under Incomplete Information [22.47189812250736]
  CoNav-Maze is a simulated environment where a robot navigates with local perception while a human operator provides guidance based on an inaccurate map.<n>We propose Information Gain Monte Carlo Tree Search (IG-MCTS) to jointly optimize autonomous movement and informative communication.<n>IG-MCTS significantly reduces communication demands and yields eye-tracking metrics indicative of lower cognitive load.
  arXiv  Detail & Related papers  (2025-02-03T22:08:04Z)
• Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots [50.02055068660255]
  Navigating urban environments poses unique challenges for robots, necessitating innovative solutions for locomotion and navigation. This work introduces a fully integrated system comprising adaptive locomotion control, mobility-aware local navigation planning, and large-scale path planning within the city. Using model-free reinforcement learning (RL) techniques and privileged learning, we develop a versatile locomotion controller. Our controllers are integrated into a large-scale urban navigation system and validated by autonomous, kilometer-scale navigation missions conducted in Zurich, Switzerland, and Seville, Spain.
  arXiv  Detail & Related papers  (2024-05-03T00:29:20Z)
• SACSoN: Scalable Autonomous Control for Social Navigation [62.59274275261392]
  We develop methods for training policies for socially unobtrusive navigation. By minimizing this counterfactual perturbation, we can induce robots to behave in ways that do not alter the natural behavior of humans in the shared space. We collect a large dataset where an indoor mobile robot interacts with human bystanders.
  arXiv  Detail & Related papers  (2023-06-02T19:07:52Z)
• Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation [92.66286342108934]
  Social navigation is the capability of an autonomous agent, such as a robot, to navigate in a'socially compliant' manner in the presence of other intelligent agents such as humans. Our dataset contains 8.7 hours, 138 trajectories, 25 miles of socially compliant, human teleoperated driving demonstrations.
  arXiv  Detail & Related papers  (2022-03-28T19:09:11Z)
• Robot Perception enables Complex Navigation Behavior via Self-Supervised Learning [23.54696982881734]
  We propose an approach to unify successful robot perception systems for active target-driven navigation tasks via reinforcement learning (RL) Our method temporally incorporates compact motion and visual perception data, directly obtained using self-supervision from a single image sequence. We demonstrate our approach on two real-world driving dataset, KITTI and Oxford RobotCar, using the new interactive CityLearn framework.
  arXiv  Detail & Related papers  (2020-06-16T07:45:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.