Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learning

• URL: http://arxiv.org/abs/2601.01818v1
• Date: Mon, 05 Jan 2026 06:14:41 GMT
• Title: Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learning
• Abstract summary: We present a language-guided scene context-aware learning framework for robust egocentric visual attention prediction.<n>We first design a context perceiver which is guided to summarize the egocentric video based on a language-based scene description.<n>We then introduce two training objectives that: 1) encourage the framework to focus on the target point-of-interest regions and 2) suppress distractions from irrelevant regions which are less likely to attract first-person attention.
• Score: 47.311695204523346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As the demand for analyzing egocentric videos grows, egocentric visual attention prediction, anticipating where a camera wearer will attend, has garnered increasing attention. However, it remains challenging due to the inherent complexity and ambiguity of dynamic egocentric scenes. Motivated by evidence that scene contextual information plays a crucial role in modulating human attention, in this paper, we present a language-guided scene context-aware learning framework for robust egocentric visual attention prediction. We first design a context perceiver which is guided to summarize the egocentric video based on a language-based scene description, generating context-aware video representations. We then introduce two training objectives that: 1) encourage the framework to focus on the target point-of-interest regions and 2) suppress distractions from irrelevant regions which are less likely to attract first-person attention. Extensive experiments on Ego4D and Aria Everyday Activities (AEA) datasets demonstrate the effectiveness of our approach, achieving state-of-the-art performance and enhanced robustness across diverse, dynamic egocentric scenarios.

Related papers

• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• Intention-Guided Cognitive Reasoning for Egocentric Long-Term Action Anticipation [52.6091162517921]
  INSIGHT is a two-stage framework for egocentric action anticipation.<n>In the first stage, INSIGHT focuses on extracting semantically rich features from hand-object interaction regions.<n>In the second stage, it introduces a reinforcement learning-based module that simulates explicit cognitive reasoning.
  arXiv  Detail & Related papers  (2025-08-03T12:52:27Z)
• HENASY: Learning to Assemble Scene-Entities for Egocentric Video-Language Model [9.762722976833581]
  Current models rely extensively on instance-level alignment between video and language modalities. We take an inspiration from human perception and explore a compositional approach for ego video representation.
  arXiv  Detail & Related papers  (2024-06-01T05:41:12Z)
• Egocentric Audio-Visual Object Localization [51.434212424829525]
  We propose a geometry-aware temporal aggregation module to handle the egomotion explicitly. The effect of egomotion is mitigated by estimating the temporal geometry transformation and exploiting it to update visual representations. It improves cross-modal localization robustness by disentangling visually-indicated audio representation.
  arXiv  Detail & Related papers  (2023-03-23T17:43:11Z)
• Bi-directional Object-context Prioritization Learning for Saliency Ranking [60.62461793691836]
  Existing approaches focus on learning either object-object or object-scene relations. We observe that spatial attention works concurrently with object-based attention in the human visual recognition system. We propose a novel bi-directional method to unify spatial attention and object-based attention for saliency ranking.
  arXiv  Detail & Related papers  (2022-03-17T16:16:03Z)
• OWL (Observe, Watch, Listen): Localizing Actions in Egocentric Video via Audiovisual Temporal Context [58.932717614439916]
  We take a deep look into the effectiveness of audio in detecting actions in egocentric videos. We propose a transformer-based model to incorporate temporal audio-visual context. Our approach achieves state-of-the-art performance on EPIC-KITCHENS-100.
  arXiv  Detail & Related papers  (2022-02-10T10:50:52Z)
• Attentional Separation-and-Aggregation Network for Self-supervised Depth-Pose Learning in Dynamic Scenes [19.704284616226552]
  Learning depth and ego-motion from unlabeled videos via self-supervision from epipolar projection can improve the robustness and accuracy of the 3D perception and localization of vision-based robots. However, the rigid projection computed by ego-motion cannot represent all scene points, such as points on moving objects, leading to false guidance in these regions. We propose an Attentional Separation-and-Aggregation Network (ASANet) which can learn to distinguish and extract the scene's static and dynamic characteristics via the attention mechanism.
  arXiv  Detail & Related papers  (2020-11-18T16:07:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.