MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model

• URL: http://arxiv.org/abs/2603.13313v1
• Date: Wed, 04 Mar 2026 08:45:25 GMT
• Title: MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model
• Abstract summary: This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS)<n>This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction.<n>By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology.
• Score: 1.552282932199974
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements have made robot navigation more intuitive by transitioning from traditional 2D displays to spatially aware Mixed Reality (MR) systems. However, current MR interfaces often rely on manual "air tap" gestures for goal placement, which can be repetitive and physically demanding, especially for beginners. This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS). This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction. By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology. Comprehensive experiments comparing MRPoS against conventional gesture-based systems demonstrate that our approach significantly reduces task completion time and workload, providing a more accessible and efficient interface. For additional material, please check: https://mertcookimg.github.io/mrpos

Related papers

• From Language to Navigation Goals: A Vision-Language Approach for Semantic Navigation of Mobile Robots Using RGB-D Perception [42.903763223989536]
  This work presents a language-driven navigation framework that enables mobile robots to interpret user requests in natural language.<n>The framework is composed of modular ROS 2 components that cooperate to transform natural language instructions into navigation actions.<n>Results show that the framework successfully interprets both direct commands and contextual requests, generates meaningful natural-language feedback, and navigates towards the desired target.
  arXiv  Detail & Related papers  (2026-07-15T09:15:21Z)
• MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes [63.12421718928034]
  We introduce MIST (the Multimodal Interactive Speech-based Tool-calling), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices.<n>We find that there is a significant gap between open-weight and closed-weight LLMs, and that even closed-weight LLMs have substantial headroom.
  arXiv  Detail & Related papers  (2026-05-07T19:57:39Z)
• CLAW: Composable Language-Annotated Whole-body Motion Generation [55.99805728566105]
  We present CLAW, a pipeline for scalable generation of language-annotated whole-body motion data.<n>CLAW composes motion primitives from a kinematic planner, parameterized by movement, heading, speed, pelvis height, and duration.<n>A low-level controller tracks these references in MuJoCo simulation, yielding physically grounded trajectories.
  arXiv  Detail & Related papers  (2026-04-13T10:02:04Z)
• MRReP: Mixed Reality-based Hand-drawn Reference Path Editing Interface for Mobile Robot Navigation [1.552282932199974]
  This paper presents MRReP, a Mixed Reality-based interface that enables users to draw a Hand-drawn Reference Path (HRP) directly on the physical floor using hand gestures.<n>The results demonstrate that MRReP enhanced path specification accuracy, usability, and perceived workload, while enabling more stable path specification in the physical environment.
  arXiv  Detail & Related papers  (2026-03-31T04:49:53Z)
• OpenFrontier: General Navigation with Visual-Language Grounded Frontiers [54.661157616245966]
  Open-world navigation requires robots to make decisions in complex everyday environments.<n>Recent advances in vision--language navigation (VLN) and vision--language--action (VLA) models enable end-to-end policies conditioned on natural language.<n>We propose OpenFrontier, a training-free navigation framework that seamlessly integrates diverse vision--language prior models.
  arXiv  Detail & Related papers  (2026-03-05T17:02:22Z)
• Direct Contact-Tolerant Motion Planning With Vision Language Models [64.48049143625063]
  Navigation in cluttered environments often requires robots to tolerate contact with movable or deformable objects.<n>We propose a direct contact-tolerant (DCT) planner, which integrates vision-language models (VLMs) into direct point perception and navigation.<n>We implement DCT in Isaac Sim and a real car-like robot, demonstrating that DCT achieves robust and efficient navigation in cluttered environments with movable obstacles.
  arXiv  Detail & Related papers  (2026-03-05T10:05:00Z)
• MOSAIC: Bridging the Sim-to-Real Gap in Generalist Humanoid Motion Tracking and Teleoperation with Rapid Residual Adaptation [50.643488406438514]
  Generalist humanoid motion trackers have recently achieved strong simulation metrics by scaling data and training, yet often remain brittle on hardware during sustained teleoperation.<n>We present MOSAIC, an open-source, full-stack system for humanoid motion tracking and whole-body teleoperation across multiple interfaces.
  arXiv  Detail & Related papers  (2026-02-09T12:35:01Z)
• RoboOmni: Proactive Robot Manipulation in Omni-modal Context [165.09049429566238]
  We introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.<n>We present RoboOmni, a framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution.<n>Experiments in simulation and real-world settings show that Robo Omni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
  arXiv  Detail & Related papers  (2025-10-27T18:49:03Z)
• An Embodied AR Navigation Agent: Integrating BIM with Retrieval-Augmented Generation for Language Guidance [8.217670177708632]
  We propose an embodied AR navigation system that supports flexible, language-driven goal retrieval and route planning.<n>The system orchestrates three language agents, Triage, Search, and Response, built on large language models.<n>A real-world user study yields a System Usability Scale (SUS) score of 80.5, indicating excellent usability.
  arXiv  Detail & Related papers  (2025-08-10T15:13:23Z)
• OpenNav: Open-World Navigation with Multimodal Large Language Models [8.41361699991122]
  Large language models (LLMs) have demonstrated strong common-sense reasoning abilities, making them promising for robotic navigation and planning tasks.<n>We aim to enable robots to interpret and decompose complex language instructions, ultimately synthesizing a sequence of trajectory points to complete diverse navigation tasks.<n>We validate our system on a Husky robot in both indoor and outdoor scenes, demonstrating its real-world robustness and applicability.
  arXiv  Detail & Related papers  (2025-07-24T02:05:28Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• Unifying Large Language Model and Deep Reinforcement Learning for Human-in-Loop Interactive Socially-aware Navigation [16.789333617628138]
  Social robot navigation planners face two major challenges: managing real-time user inputs and ensuring socially compliant behaviors.<n>We introduce SALM, an interactive, human-in-loop Socially-Aware navigation Large Language Model framework.<n>A memory mechanism archives temporal data for continuous refinement, while a multi-step graph-of-thoughts inference-based large language feedback model adaptively fuses the strengths of both planning approaches.
  arXiv  Detail & Related papers  (2024-03-22T23:12:28Z)
• Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Language Models [8.668211481067457]
  Co-NavGPT is a novel framework that integrates a Vision Language Model (VLM) as a global planner.<n>Co-NavGPT aggregates sub-maps from multiple robots with diverse viewpoints into a unified global map.<n>The VLM uses this information to assign frontiers across the robots, facilitating coordinated and efficient exploration.
  arXiv  Detail & Related papers  (2023-10-11T23:17:43Z)
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action [76.71101507291473]
  We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories. We show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data.
  arXiv  Detail & Related papers  (2022-07-10T10:41:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.