CSGaze: Context-aware Social Gaze Prediction

• URL: http://arxiv.org/abs/2511.05955v2
• Date: Fri, 14 Nov 2025 14:54:18 GMT
• Title: CSGaze: Context-aware Social Gaze Prediction
• Abstract summary: We introduce CSGaze, a context aware multimodal approach that leverages facial, scene information as complementary inputs to enhance social gaze pattern prediction.<n> Experimental results show that CSGaze performs competitively with state-of-the-art methods on GP-Static, UCO-LAEO and AVA-LAEO.<n>We provide initial explainability through generated attention scores, offering insights into the model's decision-making process.
• Score: 19.96601346116304
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A person's gaze offers valuable insights into their focus of attention, level of social engagement, and confidence. In this work, we investigate how contextual cues combined with visual scene and facial information can be effectively utilized to predict and interpret social gaze patterns during conversational interactions. We introduce CSGaze, a context aware multimodal approach that leverages facial, scene information as complementary inputs to enhance social gaze pattern prediction from multi-person images. The model also incorporates a fine-grained attention mechanism centered on the principal speaker, which helps in better modeling social gaze dynamics. Experimental results show that CSGaze performs competitively with state-of-the-art methods on GP-Static, UCO-LAEO and AVA-LAEO. Our findings highlight the role of contextual cues in improving social gaze prediction. Additionally, we provide initial explainability through generated attention scores, offering insights into the model's decision-making process. We also demonstrate our model's generalizability by testing our model on open set datasets that demonstrating its robustness across diverse scenarios.

Related papers

• Human-Inspired Social Engagement Analysis via Interpretable Mutual Visual Attention [7.72838047070244]
  We propose an interpretable computational model of social engagement inspired by psychological theories of mutual visual attention.<n>Our framework explicitly models dyadic visual attention and aggregates these cues into interpretable measures of individual and group engagement.
  arXiv  Detail & Related papers  (2026-08-25T14:05:35Z)
• SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding [10.66901115053585]
  This paper presents a novel unified framework, SAGE (Synchronized Action-Gaz-E), which integrates simultaneous recognition and anticipation of both human gaze into a single unified end-to-end trainable model model.<n>Our approach leverages a transformer-based architecture and incorporates gaze data intotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior.
  arXiv  Detail & Related papers  (2026-07-04T20:23:47Z)
• Modeling Subjective Urban Perception with Human Gaze [60.73742038237224]
  We introduce Place Pulse-Gaze, an urban perception dataset that augments street view images with synchronized eye-tracking recordings and individual perception labels.<n>We propose a Gaze-Guided Urban Perception Framework to study how gaze behavior contributes to the modeling of subjective urban perception.
  arXiv  Detail & Related papers  (2026-05-01T16:25:13Z)
• Gaze-Regularized VLMs for Ego-Centric Behavior Understanding [7.281396624646809]
  Eye gaze, encompassing fixations and saccades, provides critical insights into human intentions and future actions.<n>This study introduces a gaze-regularized framework that enhances Vision Language Models (VLMs) for egocentric behavior understanding.
  arXiv  Detail & Related papers  (2026-03-24T13:37:28Z)
• Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding [7.281396624646809]
  Eye gaze offers valuable cues about attention, short-term intent, and future actions.<n>We propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks.<n>We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze.
  arXiv  Detail & Related papers  (2025-10-24T11:33:03Z)
• Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm [36.752693539572086]
  We introduce GLARIFY, a novel method to leverage gaze information to enhance the model's effectiveness in real-world applications.<n>We analyzed hundreds of samples with the gaze modality to demonstrate the noisy nature of users' gaze patterns.<n>Experiments demonstrate that GLARIFY significantly outperforms baselines.
  arXiv  Detail & Related papers  (2025-09-26T07:02:40Z)
• Emotic Masked Autoencoder with Attention Fusion for Facial Expression Recognition [1.4374467687356276]
  This paper presents an innovative approach integrating the MAE-Face self-supervised learning (SSL) method and multi-view Fusion Attention mechanism for expression classification. We suggest easy-to-implement and no-training frameworks aimed at highlighting key facial features to determine if such features can serve as guides for the model. The efficacy of this method is validated by improvements in model performance on the Aff-wild2 dataset.
  arXiv  Detail & Related papers  (2024-03-19T16:21:47Z)
• GazeFusion: Saliency-Guided Image Generation [50.37783903347613]
  Diffusion models offer unprecedented image generation power given just a text prompt.<n>We present a saliency-guided framework to incorporate the data priors of human visual attention mechanisms into the generation process.
  arXiv  Detail & Related papers  (2024-03-16T21:01:35Z)
• A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction [11.342075103251572]
  We introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. The framework comprises of: (i) a temporal, transformer-based architecture that handles person-specific tokens capturing the gaze information related to each individual. We show that our model trained on VSGaze can address all tasks jointly, and achieves state-of-the-art results for multi-person gaze following and social gaze prediction.
  arXiv  Detail & Related papers  (2024-03-15T17:50:45Z)
• Understanding Before Recommendation: Semantic Aspect-Aware Review Exploitation via Large Language Models [53.337728969143086]
  Recommendation systems harness user-item interactions like clicks and reviews to learn their representations. Previous studies improve recommendation accuracy and interpretability by modeling user preferences across various aspects and intents. We introduce a chain-based prompting approach to uncover semantic aspect-aware interactions.
  arXiv  Detail & Related papers  (2023-12-26T15:44:09Z)
• Voila-A: Aligning Vision-Language Models with User's Gaze Attention [56.755993500556734]
  We introduce gaze information as a proxy for human attention to guide Vision-Language Models (VLMs) We propose a novel approach, Voila-A, for gaze alignment to enhance the interpretability and effectiveness of these models in real-world applications.
  arXiv  Detail & Related papers  (2023-12-22T17:34:01Z)
• Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting [74.68371461260946]
  SocialSense is a framework that induces a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. Our method surpasses existing state-of-the-art in experimental evaluations for both zero-shot and supervised settings.
  arXiv  Detail & Related papers  (2023-10-20T06:17:02Z)
• Conditioned Human Trajectory Prediction using Iterative Attention Blocks [70.36888514074022]
  We present a simple yet effective pedestrian trajectory prediction model aimed at pedestrians positions prediction in urban-like environments. Our model is a neural-based architecture that can run several layers of attention blocks and transformers in an iterative sequential fashion. We show that without explicit introduction of social masks, dynamical models, social pooling layers, or complicated graph-like structures, it is possible to produce on par results with SoTA models.
  arXiv  Detail & Related papers  (2022-06-29T07:49:48Z)
• Enhancing Social Relation Inference with Concise Interaction Graph and Discriminative Scene Representation [56.25878966006678]
  We propose an approach of textbfPRactical textbfInference in textbfSocial rtextbfElation (PRISE) It concisely learns interactive features of persons and discriminative features of holistic scenes. PRISE achieves 6.8$%$ improvement for domain classification in PIPA dataset.
  arXiv  Detail & Related papers  (2021-07-30T04:20:13Z)
• Keeping Up Appearances: Computational Modeling of Face Acts in Persuasion Oriented Discussions [2.9628298226732612]
  We propose a framework for modeling face acts in persuasion conversations. The framework reveals insights about differences in face act utilization between asymmetric roles in persuasion conversations. Using computational models, we are able to successfully identify face acts as well as predict a key conversational outcome.
  arXiv  Detail & Related papers  (2020-09-22T21:02:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.