GazeMoE: Perception of Gaze Target with Mixture-of-Experts

• URL: http://arxiv.org/abs/2603.06256v1
• Date: Fri, 06 Mar 2026 13:16:29 GMT
• Title: GazeMoE: Perception of Gaze Target with Mixture-of-Experts
• Abstract summary: Estimating human gaze target from visible images is a critical task for robots to understand human attention.<n>We propose GazeMoE, a novel end-to-end framework that selectively leverages gaze-target-related cues from a frozen foundation model.<n>GazeMoE achieves state-of-the-art performance, outperforming existing methods on challenging gaze estimation tasks.
• Score: 5.556628823471207
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Estimating human gaze target from visible images is a critical task for robots to understand human attention, yet the development of generalizable neural architectures and training paradigms remains challenging. While recent advances in pre-trained vision foundation models offer promising avenues for locating gaze targets, the integration of multi-modal cues -- including eyes, head poses, gestures, and contextual features -- demands adaptive and efficient decoding mechanisms. Inspired by Mixture-of-Experts (MoE) for adaptive domain expertise in large vision-language models, we propose GazeMoE, a novel end-to-end framework that selectively leverages gaze-target-related cues from a frozen foundation model through MoE modules. To address class imbalance in gaze target classification (in-frame vs. out-of-frame) and enhance robustness, GazeMoE incorporates a class-balancing auxiliary loss alongside strategic data augmentations, including region-specific cropping and photometric transformations. Extensive experiments on benchmark datasets demonstrate that our GazeMoE achieves state-of-the-art performance, outperforming existing methods on challenging gaze estimation tasks. The code and pre-trained models are released at https://huggingface.co/zdai257/GazeMoE

Related papers

• TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues [15.615000069929605]
  Multi-branch methods require extra supervision and annotations.<n> streamlined designs prioritize low-level visual saliency over true gaze intent.<n>We propose TextGaze, a unified cross-modal architecture that leverages a Large Vision-Language Model.
  arXiv  Detail & Related papers  (2026-07-11T05:40:45Z)
• GEM: Generative Supervision Helps Embodied Intelligence [60.28190143133428]
  Embodied Vision-Language Models (VLMs) have demonstrated impressive performance and generalization in robotics.<n>A significant gap remains between the high-level semantic focus of standard text-guided pre-training paradigms and the low-level spatial and physical knowledge critical for execution in embodied environments.<n>We introduce GEM, a Generative-supervised Embodied vision-language Model designed to bridge this divide.
  arXiv  Detail & Related papers  (2026-05-27T14:39:42Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following [67.84259349601501]
  We propose a novel training mechanism to enhance gaze reasoning in vision foundation models for gaze following.<n>Our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient.
  arXiv  Detail & Related papers  (2026-05-21T15:21:14Z)
• GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection [80.12497948980378]
  Current deepfake attribution or deepfake detection works tend to exhibit poor generalization to novel generative methods.<n>We propose a novel gaze-guided CLIP with adaptive-enhanced fine-grained language prompts for fine-grained deepfake attribution and detection (DFAD)<n>We conduct a novel and fine-grained benchmark to evaluate the DFAD performance of networks on novel generators like diffusion and flow models.<n>We introduce a gaze-aware model based on CLIP, which is devised to enhance the generalization to unseen face forgery attacks.
  arXiv  Detail & Related papers  (2026-03-31T05:59:59Z)
• Gaze-Regularized VLMs for Ego-Centric Behavior Understanding [7.281396624646809]
  Eye gaze, encompassing fixations and saccades, provides critical insights into human intentions and future actions.<n>This study introduces a gaze-regularized framework that enhances Vision Language Models (VLMs) for egocentric behavior understanding.
  arXiv  Detail & Related papers  (2026-03-24T13:37:28Z)
• Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels [10.827081942898506]
  We introduce a novel Self-Training Weakly-Supervised Gaze Estimation framework (ST-WSGE)<n>We propose the Gaze Transformer (GaT), a modality-agnostic architecture capable of simultaneously learning static and dynamic gaze information from both image and video datasets.<n>By combining 3D video datasets with 2D gaze target labels from gaze following tasks, our approach achieves the following key contributions.
  arXiv  Detail & Related papers  (2025-02-27T16:35:25Z)
• UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training [12.680014448486242]
  We propose UniGaze, leveraging large-scale in-the-wild facial datasets for gaze estimation through self-supervised pre-training.<n>Our experiments reveal that self-supervised approaches designed for semantic tasks fail when applied to gaze estimation.<n>We demonstrate that UniGaze significantly improves generalization across multiple data domains while minimizing reliance on costly labeled data.
  arXiv  Detail & Related papers  (2025-02-04T13:24:23Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• A Simple Background Augmentation Method for Object Detection with Diffusion Model [53.32935683257045]
  In computer vision, it is well-known that a lack of data diversity will impair model performance. We propose a simple yet effective data augmentation approach by leveraging advancements in generative models. Background augmentation, in particular, significantly improves the models' robustness and generalization capabilities.
  arXiv  Detail & Related papers  (2024-08-01T07:40:00Z)
• Gaze-guided Hand-Object Interaction Synthesis: Dataset and Method [61.19028558470065]
  We present GazeHOI, the first dataset to capture simultaneous 3D modeling of gaze, hand, and object interactions.<n>To tackle these issues, we propose a stacked gaze-guided hand-object interaction diffusion model, named GHO-Diffusion.<n>We also introduce HOI-Manifold Guidance during the sampling stage of GHO-Diffusion, enabling fine-grained control over generated motions.
  arXiv  Detail & Related papers  (2024-03-24T14:24:13Z)
• Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
  We develop an efficient, autoregression-based vision model on a limited dataset. We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
  arXiv  Detail & Related papers  (2024-02-07T13:41:53Z)
• Voila-A: Aligning Vision-Language Models with User's Gaze Attention [56.755993500556734]
  We introduce gaze information as a proxy for human attention to guide Vision-Language Models (VLMs) We propose a novel approach, Voila-A, for gaze alignment to enhance the interpretability and effectiveness of these models in real-world applications.
  arXiv  Detail & Related papers  (2023-12-22T17:34:01Z)
• Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention [44.10971508325032]
  We develop a novel model for zero-shot learning where gaze is predicted for never-before-searched objects. Gazeformer encodes the target using a natural language model, thus leveraging semantic similarities in scanpath prediction. It also outperforms existing target-detection models on standard gaze prediction for both target-present and target-absent search tasks.
  arXiv  Detail & Related papers  (2023-03-27T15:02:48Z)
• Scaling Vision-Language Models with Sparse Mixture of Experts [128.0882767889029]
  We show that mixture-of-experts (MoE) techniques can achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling vision-language models.
  arXiv  Detail & Related papers  (2023-03-13T16:00:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.