Gaze Target Estimation Anywhere with Concepts
- URL: http://arxiv.org/abs/2608.11367v1
- Date: Tue, 11 Aug 2026 19:23:48 GMT
- Title: Gaze Target Estimation Anywhere with Concepts
- Abstract summary: We introduce the Promptable Gaze Target Estimation (PGE) task for gaze analysis.<n>PGE conditions gaze prediction on flexible user text or visual prompts to identify a specific subject.<n>We develop a scalable data engine to generate Gaze-Co, a dataset and benchmark of 120K high-quality, prompt-annotated image pairs.<n>We also propose GazeAnywhere, the first model designed for PGE.
- Score: 28.49132281076088
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.
Related papers
- Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning [22.665947623978]
We propose a two-stage gaze estimation framework guided by object semantics.<n>Our method achieves strong performance across all benchmarks while maintaining a compact parameter size of 7.1M.
arXiv Detail & Related papers (2026-06-28T11:02:21Z) - OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
arXiv Detail & Related papers (2026-05-26T00:08:06Z) - GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection [80.12497948980378]
Current deepfake attribution or deepfake detection works tend to exhibit poor generalization to novel generative methods.<n>We propose a novel gaze-guided CLIP with adaptive-enhanced fine-grained language prompts for fine-grained deepfake attribution and detection (DFAD)<n>We conduct a novel and fine-grained benchmark to evaluate the DFAD performance of networks on novel generators like diffusion and flow models.<n>We introduce a gaze-aware model based on CLIP, which is devised to enhance the generalization to unseen face forgery attacks.
arXiv Detail & Related papers (2026-03-31T05:59:59Z) - Revisiting Salient Object Detection from an Observer-Centric Perspective [48.99721284788945]
We propose Observer-Centric Salient Object Detection (OC-SOD), where salient regions are predicted by considering not only the visual cues but also the observer-specific factors such as their preferences or intents.<n>As a result, this formulation captures the intrinsic ambiguity and diversity of human perception, enabling personalized and context-aware saliency prediction.
arXiv Detail & Related papers (2026-02-06T03:53:01Z) - Hybrid-Domain Adaptative Representation Learning for Gaze Estimation [20.422491630669885]
We present a novel Hybrid-domain Adaptative Representation Learning framework to learn robust gaze representation.<n>We propose to disentangle gaze-relevant representation from low-quality facial images by aligning features extracted from high-quality near-eye images.<n>Experiments on EyeDiap, MPIIFaceGaze, and Gaze360 datasets demonstrate that our approach achieves state-of-the-art accuracy.
arXiv Detail & Related papers (2025-11-17T10:38:50Z) - OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild [104.57404324262556]
Current 3D gaze estimation methods struggle to generalize across diverse data domains.<n>We present OmniGaze, a semi-supervised framework for 3D gaze estimation.<n>We show that OmniGaze achieves state-of-the-art performance on five datasets.
arXiv Detail & Related papers (2025-10-15T15:19:52Z) - Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders [33.26237143983192]
We address the problem of gaze target estimation, which aims to predict where a person is looking in a scene.<n>We propose Gaze-LLE, a novel transformer framework that streamlines gaze target estimation by leveraging features from a frozen DINOv2 encoder.
arXiv Detail & Related papers (2024-12-12T18:55:30Z) - Towards Pixel-Level Prediction for Gaze Following: Benchmark and Approach [27.84672974344777]
We propose a novel gaze target prediction solution named GazeSeg.<n>It can fully utilize the spatial visual field of the person as guiding information and lead to a progressively coarse-to-fine gaze target segmentation and recognition process.<n>Our approach achieves the Dice of 0.325 in gaze target segmentation and 71.7% top-5 recognition.
arXiv Detail & Related papers (2024-11-30T01:27:48Z) - LatentGaze: Cross-Domain Gaze Estimation through Gaze-Aware Analytic
Latent Code Manipulation [0.0]
We propose a gaze-aware analytic manipulation method, based on a data-driven approach with generative adversarial network inversion's disentanglement characteristics.
By utilizing GAN-based encoder-generator process, we shift the input image from the target domain to the source domain image, which a gaze estimator is sufficiently aware.
arXiv Detail & Related papers (2022-09-21T08:05:53Z) - End-to-End Human-Gaze-Target Detection with Transformers [57.00864538284686]
We propose an effective and efficient method for Human-Gaze-Target (HGT) detection, i.e., gaze following.
Our method, named Human-Gaze-Target detection TRansformer or HGTTR, streamlines the HGT detection pipeline by eliminating all other components.
The effectiveness and robustness of our proposed method are verified with extensive experiments on the two standard benchmark datasets, GazeFollowing and VideoAttentionTarget.
arXiv Detail & Related papers (2022-03-20T02:37:06Z) - Weakly-Supervised Physically Unconstrained Gaze Estimation [80.66438763587904]
We tackle the previously unexplored problem of weakly-supervised gaze estimation from videos of human interactions.
We propose a training algorithm along with several novel loss functions especially designed for the task.
We show significant improvements in (a) the accuracy of semi-supervised gaze estimation and (b) cross-domain generalization on the state-of-the-art physically unconstrained in-the-wild Gaze360 gaze estimation benchmark.
arXiv Detail & Related papers (2021-05-20T14:58:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.