Automated Detection of Mutual Gaze and Joint Attention in Dual-Camera Settings via Dual-Stream Transformers

• URL: http://arxiv.org/abs/2604.27105v1
• Date: Wed, 29 Apr 2026 18:49:34 GMT
• Title: Automated Detection of Mutual Gaze and Joint Attention in Dual-Camera Settings via Dual-Stream Transformers
• Abstract summary: Mutual gaze (MG) and joint attention (JA) is critical in developmental psychology but traditionally relies on labor-intensive manual coding.<n>We propose a highly efficient dual-stream Transformer architecture for detecting MG and JA from synchronized dual-camera recordings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Analyzing mutual gaze (MG) and joint attention (JA) is critical in developmental psychology but traditionally relies on labor-intensive manual coding. Automating this process in multi-camera laboratory settings is computationally challenging due to complex cross-camera relational dynamics. In this paper, we propose a highly efficient dual-stream Transformer architecture for detecting MG and JA from synchronized dual-camera recordings. Our approach leverages frozen gaze-aware backbones (GazeLLE) to extract rich visual priors, combined with a custom token fusion mechanism to map the spatial and semantic relationships between interacting dyads. Evaluated on an ecologically valid dataset of caregiver-infant interactions, our model exhibits good performance, significantly outperforming both a convolutional baseline and a state-of-the-art multimodal Large Language Model (LLM). By open-sourcing our model and pre-trained weights, we provide behavioral scientists with a scalable tool that can be fine-tuned to diverse laboratory environments, effectively bridging the gap between computational modeling and applied interaction research.

Related papers

• Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis [55.78822230833247]
  We present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis.<n>Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations.<n>We propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimize interaction reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2026-08-20T17:51:48Z)
• An Empirical Study of Interaction Smells in Multi-Turn Human-LLM Collaborative Code Generation [10.568269273364448]
  Large Language Models (LLMs) have revolutionized code generation, evolving from static tools into dynamic conversational interfaces.<n>LLMs exhibit remarkable proficiency in generating standalone code snippets, but struggle to maintain contextual consistency during extended interactions.<n>Existing benchmarks primarily emphasize the functional correctness of the final output, overlooking latent quality issues within the interaction process itself, which we term Interaction Smells.
  arXiv  Detail & Related papers  (2026-03-10T14:12:18Z)
• DualGazeNet: A Biologically Inspired Dual-Gaze Query Network for Salient Object Detection [52.32976488996896]
  We introduce DualGazeNet, a pure Transformer framework for salient object detection.<n>Experiments on five RGB benchmarks show that DualGazeNet consistently surpasses 25 state-of-the-art CNN- and Transformer-based methods.
  arXiv  Detail & Related papers  (2025-11-24T08:08:22Z)
• Explicit Multimodal Graph Modeling for Human-Object Interaction Detection [11.15526365654911]
  Graph Neural Networks (GNNs) are inherently better suited for this task, as they explicitly model the relationships between human-object pairs.<n>We propose textbfMultimodal textbfGraph textbfNetwork textbfModeling (MGNM) that leverages GNN-based relational structures to enhance HOI detection.
  arXiv  Detail & Related papers  (2025-09-16T01:17:49Z)
• Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognition [30.016692048849226]
  Micro-gestures are unconsciously performed body gestures that can convey the emotion states of humans.<n>We propose to recognize the emotion states based on the micro-gestures by reconstructing the behavior patterns with a hypergraph-enhanced Transformer.<n>The proposed method is evaluated on two publicly available datasets, namely iMiGUE and SMG.
  arXiv  Detail & Related papers  (2025-07-20T08:27:56Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR [50.435592120607815]
  Scene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR) Previous works have primarily relied on multi-stage learning, where the generated semantic scene graphs depend on intermediate processes with pose estimation and object detection. In this study, we introduce a novel single-stage bi-modal transformer framework for SGG in the OR, termed S2Former-OR.
  arXiv  Detail & Related papers  (2024-02-22T11:40:49Z)
• Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection [116.21529970404653]
  We introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Our approach achieves competitive performance when compared to state-of-the-art HOI methods.
  arXiv  Detail & Related papers  (2023-11-03T07:25:57Z)
• A Multi-label Classification Approach to Increase Expressivity of EMG-based Gesture Recognition [4.701158597171363]
  The aim of this study is to efficiently increase the expressivity of surface electromyography-based (sEMG) gesture recognition systems. We use a problem transformation approach, in which actions were subset into two biomechanically independent components.
  arXiv  Detail & Related papers  (2023-09-13T20:21:41Z)
• Cascaded Human-Object Interaction Recognition [175.60439054047043]
  We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
  arXiv  Detail & Related papers  (2020-03-09T17:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.