Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following

• URL: http://arxiv.org/abs/2605.22607v1
• Date: Thu, 21 May 2026 15:21:14 GMT
• Title: Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following
• Abstract summary: We propose a novel training mechanism to enhance gaze reasoning in vision foundation models for gaze following.<n>Our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient.
• Score: 67.84259349601501
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaze following requires both scene understanding and gaze reasoning to localize the gaze target of an in-scene person. Recently, vision foundation models (VFMs) have demonstrated strong performance on this task, enabling simpler architectures while outperforming prior methods. However, we observe a key limitation of VFM-based approaches: while VFMs substantially improve scene understanding, they contribute little to gaze reasoning. As a result, existing methods often rely on semantically salient objects rather than true gaze cues, leading to degraded performance when targets are not salient. To address this, we propose a novel training mechanism to enhance gaze reasoning in VFMs for gaze following. Our method includes: (1) a head-conditioned local LoRA, which enables localized adaptation to preserve scene token learning while improving head token learning for gaze reasoning; and (2) an out-of-cone penalty, which injects gaze cues into head tokens while aligning them with scene tokens. Experiments on the GazeFollow and VAT datasets demonstrate that our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient. Our findings offer valuable insights for advancing future gaze following research. We will release the code once the paper is accepted.

Related papers

• From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation [6.812620443565403]
  This work introduces the first training-free Gaze Target Agent (GTA) for gaze-guided reasoning across tasks such as gaze target prediction, attention localization, and object identification.<n>This is achieved by leveraging pretrained vision-language models, augmenting them with visually guided prompts, and employing a memory-based retrieval strategy for high-uncertainty samples.
  arXiv  Detail & Related papers  (2026-09-05T17:54:58Z)
• Visually-Guided Policy Optimization for Multimodal Reasoning [60.035908460318126]
  We propose Visually-Guided Policy Optimization (VGPO) to reinforce visual focus during policy optimization.<n> VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues.<n>We show that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks.
  arXiv  Detail & Related papers  (2026-04-10T14:22:38Z)
• GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection [80.12497948980378]
  Current deepfake attribution or deepfake detection works tend to exhibit poor generalization to novel generative methods.<n>We propose a novel gaze-guided CLIP with adaptive-enhanced fine-grained language prompts for fine-grained deepfake attribution and detection (DFAD)<n>We conduct a novel and fine-grained benchmark to evaluate the DFAD performance of networks on novel generators like diffusion and flow models.<n>We introduce a gaze-aware model based on CLIP, which is devised to enhance the generalization to unseen face forgery attacks.
  arXiv  Detail & Related papers  (2026-03-31T05:59:59Z)
• Gaze-Regularized VLMs for Ego-Centric Behavior Understanding [7.281396624646809]
  Eye gaze, encompassing fixations and saccades, provides critical insights into human intentions and future actions.<n>This study introduces a gaze-regularized framework that enhances Vision Language Models (VLMs) for egocentric behavior understanding.
  arXiv  Detail & Related papers  (2026-03-24T13:37:28Z)
• Rethinking VLMs for Image Forgery Detection and Localization [55.32700985102152]
  We study how to fully leverage vision-language models (VLMs) to assist the image forgery detection and localization (IFDL) task.<n>Building on these findings, we propose a new IFDL pipeline named IFDL-VLM.<n>The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.
  arXiv  Detail & Related papers  (2026-03-13T12:21:31Z)
• StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos [128.45606644157]
  StreamGaze is the first benchmark to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos.<n>We develop a gaze-video QA generation pipeline that aligns egocentric videos with raw gaze trajectories.<n>We observe substantial performance gaps between state-of-the-art MLLMs and human performance.
  arXiv  Detail & Related papers  (2025-12-01T14:15:44Z)
• GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding [5.94301570835109]
  This paper introduces GazeVLM, a novel Vision-Language Model (VLM) for multi-task gaze understanding in images.<n>It addresses person detection, gaze target detection, and gaze object identification.<n>GazeVLM represents, to our knowledge, the first application of a VLM to these combined tasks, allowing for selective execution of each task.
  arXiv  Detail & Related papers  (2025-11-09T12:07:40Z)
• Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding [7.281396624646809]
  Eye gaze offers valuable cues about attention, short-term intent, and future actions.<n>We propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks.<n>We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze.
  arXiv  Detail & Related papers  (2025-10-24T11:33:03Z)
• LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation [12.903711441941663]
  The ability of gaze estimation models to generalize is often significantly hindered by various factors unrelated to gaze. We propose a novel approach, reframing the gaze estimation task as a vision-language alignment issue. Our proposed framework, named Language-Guided Gaze Estimation (LG-Gaze), learns continuous and geometry-sensitive features for gaze estimation benefit from the rich prior knowledges of vision-language models.
  arXiv  Detail & Related papers  (2024-11-13T13:46:15Z)
• Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following [10.91834567383105]
  Contextual cues related to a person's pose and interactions with objects can provide valuable information for gaze following. We evaluate Vision-Language Models (VLMs) for extracting a wide array of contextual cues to improve gaze following performance. Using the entire image along with an ellipse drawn around the target person is the most effective strategy for visual prompting.
  arXiv  Detail & Related papers  (2024-06-06T09:41:39Z)
• Weakly-Supervised Physically Unconstrained Gaze Estimation [80.66438763587904]
  We tackle the previously unexplored problem of weakly-supervised gaze estimation from videos of human interactions. We propose a training algorithm along with several novel loss functions especially designed for the task. We show significant improvements in (a) the accuracy of semi-supervised gaze estimation and (b) cross-domain generalization on the state-of-the-art physically unconstrained in-the-wild Gaze360 gaze estimation benchmark.
  arXiv  Detail & Related papers  (2021-05-20T14:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.