A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction

• URL: http://arxiv.org/abs/2403.10511v1
• Date: Fri, 15 Mar 2024 17:50:45 GMT
• Title: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction
• Abstract summary: We introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. The framework comprises of: (i) a temporal, transformer-based architecture that handles person-specific tokens capturing the gaze information related to each individual. We show that our model trained on VSGaze can address all tasks jointly, and achieves state-of-the-art results for multi-person gaze following and social gaze prediction.
• Score: 11.342075103251572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaze following and social gaze prediction are fundamental tasks providing insights into human communication behaviors, intent, and social interactions. Most previous approaches addressed these tasks separately, either by designing highly specialized social gaze models that do not generalize to other social gaze tasks or by considering social gaze inference as an ad-hoc post-processing of the gaze following task. Furthermore, the vast majority of gaze following approaches have proposed static models that can handle only one person at a time, therefore failing to take advantage of social interactions and temporal dynamics. In this paper, we address these limitations and introduce a novel framework to jointly predict the gaze target and social gaze label for all people in the scene. The framework comprises of: (i) a temporal, transformer-based architecture that, in addition to image tokens, handles person-specific tokens capturing the gaze information related to each individual; (ii) a new dataset, VSGaze, that unifies annotation types across multiple gaze following and social gaze datasets. We show that our model trained on VSGaze can address all tasks jointly, and achieves state-of-the-art results for multi-person gaze following and social gaze prediction.

Related papers

• Gaze-Anchored Social Net: Decoding Implicit Relations via Joint Modeling [42.54041031276457]
  We propose ANCHOR, a target-centric paradigm designed to decode gaze-anchored social intent.<n>Our approach surfaces these dependencies as the latent structural scaffolding of gaze behavior.<n>We validate our framework on an extended benchmark featuring dense multi-person annotations and novel social influence rankings.
  arXiv  Detail & Related papers  (2026-07-24T18:39:14Z)
• SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding [10.66901115053585]
  This paper presents a novel unified framework, SAGE (Synchronized Action-Gaz-E), which integrates simultaneous recognition and anticipation of both human gaze into a single unified end-to-end trainable model model.<n>Our approach leverages a transformer-based architecture and incorporates gaze data intotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior.
  arXiv  Detail & Related papers  (2026-07-04T20:23:47Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• CSGaze: Context-aware Social Gaze Prediction [19.96601346116304]
  We introduce CSGaze, a context aware multimodal approach that leverages facial, scene information as complementary inputs to enhance social gaze pattern prediction.<n> Experimental results show that CSGaze performs competitively with state-of-the-art methods on GP-Static, UCO-LAEO and AVA-LAEO.<n>We provide initial explainability through generated attention scores, offering insights into the model's decision-making process.
  arXiv  Detail & Related papers  (2025-11-08T10:07:45Z)
• MINGLE: VLMs for Semantically Complex Region Detection in Urban Scenes [49.89767522399176]
  Group-level social interactions in public spaces are crucial for urban planning.<n>We introduce a social group region detection task, which requires inferring and spatially grounding visual regions defined by interpersonal relations.<n>We propose MINGLE, a modular three-stage pipeline that integrates human detection and depth estimation, VLM-based reasoning to classify pairwise social affiliation, and a lightweight spatial aggregation algorithm to localize socially connected groups.<n>We present a new dataset of 100K urban street-view images annotated with bounding boxes and labels for both individuals and socially interacting groups.
  arXiv  Detail & Related papers  (2025-09-16T19:31:40Z)
• GazeDETR: Gaze Detection using Disentangled Head and Gaze Representations [14.82916312780764]
  We propose GazeDETR, a novel end-to-end architecture with two disentangled decoders.<n>Our proposed architecture achieves state-of-the-art results on the GazeFollow, VideoAttentionTarget and ChildPlay datasets.
  arXiv  Detail & Related papers  (2025-08-18T14:41:18Z)
• SocialGesture: Delving into Multi-person Gesture Understanding [24.407821757676615]
  SocialGesture is the first large-scale dataset specifically designed for multi-person gesture analysis. SocialGesture features a diverse range of natural scenarios and supports multiple gesture analysis tasks.
  arXiv  Detail & Related papers  (2025-04-03T03:21:06Z)
• Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses [11.545286742778977]
  We first report a comprehensive analysis of eye-body coordination in various human-object and human-human interaction activities. We then present Pose2Gaze, a eye-body coordination model that uses a convolutional neural network to extract features from head direction and full-body poses.
  arXiv  Detail & Related papers  (2023-12-19T10:55:46Z)
• Sharingan: A Transformer-based Architecture for Gaze Following [14.594691605523005]
  We introduce a novel transformer-based architecture for 2D gaze prediction. This paper achieves state-of-the-art results on the GazeFollow and VideoTarget datasets.
  arXiv  Detail & Related papers  (2023-10-01T23:14:54Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)
• GazeOnce: Real-Time Multi-Person Gaze Estimation [18.16091280655655]
  Appearance-based gaze estimation aims to predict the 3D eye gaze direction from a single image. Recent deep learning-based approaches have demonstrated excellent performance, but cannot output multi-person gaze in real time. We propose GazeOnce, which is capable of simultaneously predicting gaze directions for multiple faces in an image.
  arXiv  Detail & Related papers  (2022-04-20T14:21:47Z)
• Didn't see that coming: a survey on non-verbal social human behavior forecasting [47.99589136455976]
  Non-verbal social human behavior forecasting has increasingly attracted the interest of the research community in recent years. Its direct applications to human-robot interaction and socially-aware human motion generation make it a very attractive field. We define the behavior forecasting problem for multiple interactive agents in a generic way that aims at unifying the fields of social signals prediction and human motion forecasting.
  arXiv  Detail & Related papers  (2022-03-04T18:25:30Z)
• SSAGCN: Social Soft Attention Graph Convolution Network for Pedestrian Trajectory Prediction [59.064925464991056]
  We propose one new prediction model named Social Soft Attention Graph Convolution Network (SSAGCN) SSAGCN aims to simultaneously handle social interactions among pedestrians and scene interactions between pedestrians and environments. Experiments on public available datasets prove the effectiveness of SSAGCN and have achieved state-of-the-art results.
  arXiv  Detail & Related papers  (2021-12-05T01:49:18Z)
• Enhancing Social Relation Inference with Concise Interaction Graph and Discriminative Scene Representation [56.25878966006678]
  We propose an approach of textbfPRactical textbfInference in textbfSocial rtextbfElation (PRISE) It concisely learns interactive features of persons and discriminative features of holistic scenes. PRISE achieves 6.8$%$ improvement for domain classification in PIPA dataset.
  arXiv  Detail & Related papers  (2021-07-30T04:20:13Z)
• Weakly-Supervised Physically Unconstrained Gaze Estimation [80.66438763587904]
  We tackle the previously unexplored problem of weakly-supervised gaze estimation from videos of human interactions. We propose a training algorithm along with several novel loss functions especially designed for the task. We show significant improvements in (a) the accuracy of semi-supervised gaze estimation and (b) cross-domain generalization on the state-of-the-art physically unconstrained in-the-wild Gaze360 gaze estimation benchmark.
  arXiv  Detail & Related papers  (2021-05-20T14:58:52Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.