An Augmented Reality Brain-Robot Interface for Generalist Robot Arm Manipulation

• URL: http://arxiv.org/abs/2606.16413v1
• Date: Mon, 15 Jun 2026 08:50:43 GMT
• Title: An Augmented Reality Brain-Robot Interface for Generalist Robot Arm Manipulation
• Abstract summary: We present an AR brain-robot interface (BRI) that combines gaze-based object selection with motor imagery action control.<n>Our system uses eye-tracking for intuitive object targeting and context-aware visual overlays to guide the user through tasks.
• Score: 0.32326259807823027
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The integration of augmented reality (AR) and EEG-based brain-computer interfaces (BCIs) offers a promising path for enabling intuitive control of robots for assistive purposes. However, existing AR brain-robot interface (BRI) systems are often constrained to task-specific structures, limiting their utility in real-world environments. We present an AR BRI designed for generalist robot arm manipulation that combines gaze-based object selection with motor imagery action control. Our system uses eye-tracking for intuitive object targeting and context-aware visual overlays ("Place" and "Use") to guide the user through tasks within a shared autonomy framework. We evaluated the interface through a feasibility study with 18 healthy participants performing three multi-step activities of daily living: drinking, using a drawer, and operating an oven. Our results demonstrate that this interaction paradigm enables effective sequential task execution and high user engagement, achieving a "Good" usability rating (SUS > 70). These findings support the feasibility of the proposed interaction paradigm for complex BCI-driven robotic assistance, and motivate future evaluation with the intended target population. Project website: https://ar-bri-manip.github.io/.

Related papers

• A Glimpse into Long-term Physical Coexistence with Intelligent Robots [12.275897522668858]
  PHILIA is a multi-robot agent built around a robot gateway abstraction.<n>It decouples low-frequency, high-semantic agent reasoning from high-frequency, low-level robot execution.<n>We present representative use cases in which agent memory and scene understanding are grounded in robot actions.
  arXiv  Detail & Related papers  (2026-07-13T10:40:32Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• Interface-Aware Trajectory Reconstruction of Limited Demonstrations for Robot Learning [0.4199844472131921]
  We present a trajectory reconstruction algorithm that lifts demonstrations into the robot's full control space.<n>We evaluate our approach using real-world demonstrations of ADL-inspired tasks performed via a 2-D joystick and 1-D sip-and-puff control interface.
  arXiv  Detail & Related papers  (2026-02-26T18:01:25Z)
• Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models [5.731055678884616]
  Eye gaze offers a fast, non-intrusive, and intent-rich input modality, making it an attractive channel for conveying user goals.<n>We present GAMMA, a system that leverages ego-centric gaze tracking and a vision-language model to infer user intent and autonomously execute robotic manipulation tasks.<n>Results demonstrate that GAMMA provides robust, intuitive, and generalizable control, highlighting the potential of combining foundation models and gaze for natural and scalable robot autonomy.
  arXiv  Detail & Related papers  (2026-01-08T19:33:03Z)
• Interpretable Robot Control via Structured Behavior Trees and Large Language Models [0.14990005092937678]
  This paper presents a novel framework that bridges natural language understanding and robotic execution.<n>The proposed approach is practical in real-world scenarios, with an average cognition-to-execution accuracy of approximately 94%.
  arXiv  Detail & Related papers  (2025-08-13T08:53:13Z)
• Recognizing Actions from Robotic View for Natural Human-Robot Interaction [52.00935005918032]
  Natural Human-Robot Interaction (N-HRI) requires robots to recognize human actions at varying distances and states, regardless of whether the robot itself is in motion or stationary.<n>Existing benchmarks for N-HRI fail to address the unique complexities in N-HRI due to limited data, modalities, task categories, and diversity of subjects and environments.<n>We introduce (Action from Robotic View) a large-scale dataset for perception-centric robotic views prevalent in mobile service robots.
  arXiv  Detail & Related papers  (2025-07-30T09:48:34Z)
• Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models [53.22792173053473]
  We introduce an interactive robotic manipulation framework called Polaris. Polaris integrates perception and interaction by utilizing GPT-4 alongside grounded vision models. We propose a novel Synthetic-to-Real (Syn2Real) pose estimation pipeline.
  arXiv  Detail & Related papers  (2024-08-15T06:40:38Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
  This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system. We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration. We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
  arXiv  Detail & Related papers  (2023-08-30T11:35:21Z)
• A ROS Architecture for Personalised HRI with a Bartender Social Robot [61.843727637976045]
  BRILLO project has the overall goal of creating an autonomous robotic bartender that can interact with customers while accomplishing its bartending tasks. We present the developed three-layers ROS architecture integrating a perception layer managing the processing of different social signals, a decision-making layer for handling multi-party interactions, and an execution layer controlling the behaviour of a complex robot composed of arms and a face.
  arXiv  Detail & Related papers  (2022-03-13T11:33:06Z)
• Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly [18.563562557565483]
  We propose the framework "Scene Editing as Teleoperation" (SEaT) Instead of controlling the robot, users focus on specifying the task's goal. A user can perform teleoperation without any expert knowledge of the robot hardware.
  arXiv  Detail & Related papers  (2021-10-09T04:22:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.