Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models

• URL: http://arxiv.org/abs/2601.05336v1
• Date: Thu, 08 Jan 2026 19:33:03 GMT
• Title: Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models
• Abstract summary: Eye gaze offers a fast, non-intrusive, and intent-rich input modality, making it an attractive channel for conveying user goals.<n>We present GAMMA, a system that leverages ego-centric gaze tracking and a vision-language model to infer user intent and autonomously execute robotic manipulation tasks.<n>Results demonstrate that GAMMA provides robust, intuitive, and generalizable control, highlighting the potential of combining foundation models and gaze for natural and scalable robot autonomy.
• Score: 5.731055678884616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing intuitive interfaces for robotic control remains a central challenge in enabling effective human-robot interaction, particularly in assistive care settings. Eye gaze offers a fast, non-intrusive, and intent-rich input modality, making it an attractive channel for conveying user goals. In this work, we present GAMMA (Gaze Assisted Manipulation for Modular Autonomy), a system that leverages ego-centric gaze tracking and a vision-language model to infer user intent and autonomously execute robotic manipulation tasks. By contextualizing gaze fixations within the scene, the system maps visual attention to high-level semantic understanding, enabling skill selection and parameterization without task-specific training. We evaluate GAMMA on a range of table-top manipulation tasks and compare it against baseline gaze-based control without reasoning. Results demonstrate that GAMMA provides robust, intuitive, and generalizable control, highlighting the potential of combining foundation models and gaze for natural and scalable robot autonomy. Project website: https://gamma0.vercel.app/

Related papers

• An Augmented Reality Brain-Robot Interface for Generalist Robot Arm Manipulation [0.32326259807823027]
  We present an AR brain-robot interface (BRI) that combines gaze-based object selection with motor imagery action control.<n>Our system uses eye-tracking for intuitive object targeting and context-aware visual overlays to guide the user through tasks.
  arXiv  Detail & Related papers  (2026-06-15T08:50:43Z)
• Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation [24.026684861369386]
  Vision-Language-Action (VLA) models have recently shown strong potential for robot learning by following language instructions.<n>It is difficult to describe which exact object to interact with among similar candidates, where to act on the object, or how the target may change during execution.<n>We propose Gaze2Act, a novel VLA framework that leverages human gaze as a dynamic and intuitive intent signal for complex interactive manipulation.
  arXiv  Detail & Related papers  (2026-05-28T17:37:16Z)
• ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration [51.69384671495837]
  ActiveGlasses is a system for learning robot manipulation from ego-centric human demonstrations with active vision.<n>A stereo camera mounted on smart glasses serves as the sole perception device for both data collection and policy inference.<n>To enable zero-transfer, we extract object trajectories from demonstrations and use an object-centric point-cloud policy to jointly predict manipulation and head movement.
  arXiv  Detail & Related papers  (2026-04-09T17:59:08Z)
• HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations [15.551927664158695]
  Whole-Body Mobile Manipulation Interface (HoMMI) learns whole-body mobile manipulation directly from robot-free human demonstrations.<n>We augment UMI interfaces with egocentric sensing to capture the global context required for mobile manipulation.<n>We explicitly bridge this gap with a cross-embodiment hand-eye policy design.
  arXiv  Detail & Related papers  (2026-03-03T18:36:49Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers [2.736848514829367]
  Human vision is a highly active process driven by gaze, which directs attention to task-relevant regions through foveation.<n>In this work, we explore how incorporating human-like active gaze into robotic policies can enhance efficiency and robustness.<n>We develop GIAVA, a robot vision system that emulates human head and neck movement, and gaze adjustment for foveated processing.
  arXiv  Detail & Related papers  (2025-07-21T17:44:10Z)
• Humanoid-VLA: Towards Universal Humanoid Control with Visual Integration [28.825612240280822]
  We propose a novel framework that integrates language understanding, egocentric scene perception, and motion control, enabling universal humanoid control.<n>Humanoid-VLA begins with language-motion pre-alignment using non-egocentric human motion datasets paired with textual descriptions.<n>We then incorporate egocentric visual context through a parameter efficient video-conditioned fine-tuning, enabling context-aware motion generation.
  arXiv  Detail & Related papers  (2025-02-20T18:17:11Z)
• Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models [53.22792173053473]
  We introduce an interactive robotic manipulation framework called Polaris. Polaris integrates perception and interaction by utilizing GPT-4 alongside grounded vision models. We propose a novel Synthetic-to-Real (Syn2Real) pose estimation pipeline.
  arXiv  Detail & Related papers  (2024-08-15T06:40:38Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance [71.36749876465618]
  We describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples. experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world.
  arXiv  Detail & Related papers  (2022-12-19T22:50:40Z)
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? [54.442692221567796]
  Task specification is critical for engagement of non-expert end-users and adoption of personalized robots. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
  arXiv  Detail & Related papers  (2022-04-23T19:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.