Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs

• URL: http://arxiv.org/abs/2603.06697v1
• Date: Thu, 05 Mar 2026 02:12:49 GMT
• Title: Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs
• Abstract summary: Vision--language models (VLMs) process images as visual tokens, yet their intermediate reasoning is often carried out in text.<n>We use eye-gaze as supervision to guide VLM reasoning by introducing a small set of dedicated gaze tokens.<n>These tokens are trained to predict gaze-selected image patch indices in temporal order, encouraging the model to follow human-like evidence acquisition and integration.
• Score: 24.510023374770316
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision--language models (VLMs) process images as visual tokens, yet their intermediate reasoning is often carried out in text, which can be suboptimal for visually grounded radiology tasks. Radiologists instead diagnose via sequential visual search; eye-tracking captures this process as time-ordered gaze trajectories that reveal how evidence is acquired over time. We use eye-gaze as supervision to guide VLM reasoning by introducing a small set of dedicated gaze tokens. These tokens are trained to predict gaze-selected image patch indices in temporal order, encouraging the model to follow human-like evidence acquisition and integration. Experiments on MIMIC-EYE and multiple external zero-shot benchmarks show consistent gains over baselines, achieving state-of-the-art in-domain performance and improved out-of-domain robustness. These results highlight temporally ordered gaze as an effective supervision signal for learning visually grounded medical reasoning.

Related papers

• LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppression [50.454633334898396]
  Large vision-language models (LVLMs) exhibit strong reasoning ability but suffer from visual forgetting during long-horizon decoding.<n>We propose LASER, a post-training framework that regulates both the visual attention trajectory and intra-visual token attention distribution during reasoning.<n>Experiments on eight benchmark datasets demonstrate that LASER consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-07-02T04:59:56Z)
• Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following [67.84259349601501]
  We propose a novel training mechanism to enhance gaze reasoning in vision foundation models for gaze following.<n>Our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient.
  arXiv  Detail & Related papers  (2026-05-21T15:21:14Z)
• Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs [66.15429821530503]
  Persistent Visual Memory is a lightweight learnable module designed to strengthen sustained, on-demand access to visual evidence.<n>Experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead.<n>In-depth analysis reveals that PVM shows improved robustness in longer generations and accelerates internal prediction convergence.
  arXiv  Detail & Related papers  (2026-05-01T17:54:37Z)
• Visually-Guided Policy Optimization for Multimodal Reasoning [60.035908460318126]
  We propose Visually-Guided Policy Optimization (VGPO) to reinforce visual focus during policy optimization.<n> VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues.<n>We show that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks.
  arXiv  Detail & Related papers  (2026-04-10T14:22:38Z)
• Improving Visual Reasoning with Iterative Evidence Refinement [41.18410988040976]
  Vision language models (VLMs) are increasingly capable of reasoning over images.<n>But robust visual reasoning often requires re-grounding intermediate steps in the underlying visual evidence.<n>We propose an end-to-end self-revisit framework, SIEVE, that trains models to re-engage image evidence through internal representations.
  arXiv  Detail & Related papers  (2026-03-14T21:08:21Z)
• Latent Implicit Visual Reasoning [59.39913238320798]
  We propose a task-agnostic mechanism that trains LMMs to discover and use visual reasoning tokens without explicit supervision.<n>Our approach outperforms direct fine-tuning and achieves state-of-the-art results on a diverse range of vision-centric tasks.
  arXiv  Detail & Related papers  (2025-12-24T14:59:49Z)
• Think Twice to See More: Iterative Visual Reasoning in Medical VLMs [21.083636394814217]
  We introduce ViTAR, a framework that emulates the iterative reasoning process of human experts through a cognitive chain of "think-act-rethink-answer"<n>ViTAR treats medical images as interactive objects, enabling models to engage multi-step visual reasoning.
  arXiv  Detail & Related papers  (2025-10-11T06:39:57Z)
• Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence [69.86946427928511]
  We investigate the internal mechanisms driving hallucination in large vision-language models (LVLMs)<n>We introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context.<n>We propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads.
  arXiv  Detail & Related papers  (2024-12-18T15:29:30Z)
• GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph [32.1234295417225]
  We propose a context-aware Gaze EstiMation (GEM) network that utilizes eye gaze data collected from radiologists to simulate their visual search behavior patterns. It consists of a context-awareness module, visual behavior graph construction, and visual behavior matching. Experiments on four publicly available datasets demonstrate the superiority of GEM over existing methods.
  arXiv  Detail & Related papers  (2024-08-10T09:46:25Z)
• Bootstrapping Vision-language Models for Self-supervised Remote Physiological Measurement [26.480515954528848]
  We propose a novel framework that successfully integrates popular vision-language models into a remote physiological measurement task.<n>We develop a series of generative and contrastive learning mechanisms to optimize the framework.<n>Our method for the first time adapts VLMs to digest and align the frequency-related knowledge in vision and text modalities.
  arXiv  Detail & Related papers  (2024-07-11T13:45:50Z)
• GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification [9.266556662553345]
  We propose a novel gaze-guided graph neural network (GNN), GazeGNN, to leverage raw eye-gaze data without being converted into visual attention maps (VAMs) We develop a real-time, real-world, end-to-end disease classification algorithm for the first time in the literature.
  arXiv  Detail & Related papers  (2023-05-29T17:01:54Z)
• A Deep Learning Approach for the Segmentation of Electroencephalography Data in Eye Tracking Applications [56.458448869572294]
  We introduce DETRtime, a novel framework for time-series segmentation of EEG data. Our end-to-end deep learning-based framework brings advances in Computer Vision to the forefront. Our model generalizes well in the task of EEG sleep stage segmentation.
  arXiv  Detail & Related papers  (2022-06-17T10:17:24Z)
• Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis [54.60796004113496]
  We demonstrate that the eye movement of radiologists reading medical images can be a new form of supervision to train the DNN-based computer-aided diagnosis (CAD) system. We record the tracks of the radiologists' gaze when they are reading images. The gaze information is processed and then used to supervise the DNN's attention via an Attention Consistency module.
  arXiv  Detail & Related papers  (2022-04-06T08:31:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.