Improving Keystep Recognition in Ego-Video via Dexterous Focus

• URL: http://arxiv.org/abs/2506.00827v1
• Date: Sun, 01 Jun 2025 04:22:02 GMT
• Title: Improving Keystep Recognition in Ego-Video via Dexterous Focus
• Authors: Zachary Chavis, Stephen J. Guy, Hyun Soo Park,
• Abstract summary: We address the challenge of understanding human activities from an egocentric perspective.<n>We propose a framework that seeks to address these challenges by restricting the ego-video input to a stabilized, hand-focused video.
• Score: 18.14234312389889
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we address the challenge of understanding human activities from an egocentric perspective. Traditional activity recognition techniques face unique challenges in egocentric videos due to the highly dynamic nature of the head during many activities. We propose a framework that seeks to address these challenges in a way that is independent of network architecture by restricting the ego-video input to a stabilized, hand-focused video. We demonstrate that this straightforward video transformation alone outperforms existing egocentric video baselines on the Ego-Exo4D Fine-Grained Keystep Recognition benchmark without requiring any alteration of the underlying model infrastructure.

Related papers

• Fine-grained Spatiotemporal Grounding on Egocentric Videos [13.319346673043286]
  We introduce EgoMask, the first pixel-level benchmark for fine-temporal grounding in egocentric videos.<n>EgoMask is constructed by our proposed automatic annotation pipeline, which annotates referring expressions and object masks.<n>We also create EgoMask-Train, a large-scale training dataset to facilitate model development.
  arXiv  Detail & Related papers  (2025-08-01T10:53:27Z)
• EgoM2P: Egocentric Multimodal Multitask Pretraining [55.259234688003545]
  Building large-scale egocentric multimodal and multitask models presents unique challenges.<n> EgoM2P is a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding.<n>We will fully open-source EgoM2P to support the community and advance egocentric vision research.
  arXiv  Detail & Related papers  (2025-06-09T15:59:25Z)
• Object-Shot Enhanced Grounding Network for Egocentric Video [60.97916755629796]
  We propose OSGNet, an Object-Shot enhanced Grounding Network for egocentric video.<n>Specifically, we extract object information from videos to enrich video representation.<n>We analyze the frequent shot movements inherent to egocentric videos, leveraging these features to extract the wearer's attention information.
  arXiv  Detail & Related papers  (2025-05-07T09:20:12Z)
• EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation [30.350824860817536]
  EgoVid-5M is the first high-quality dataset curated for egocentric video generation. We introduce EgoDreamer, which is capable of generating egocentric videos driven simultaneously by action descriptions and kinematic control signals.
  arXiv  Detail & Related papers  (2024-11-13T07:05:40Z)
• EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars [56.56236652774294]
  We propose a person-specific egocentric telepresence approach, which jointly models the photoreal digital avatar while also driving it from a single egocentric video. Our experiments demonstrate a clear step towards egocentric and photoreal telepresence as our method outperforms baselines as well as competing methods.
  arXiv  Detail & Related papers  (2024-09-22T22:50:27Z)
• AMEGO: Active Memory from long EGOcentric videos [26.04157621755452]
  We introduce AMEGO, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. Inspired by the human's ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video. This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content.
  arXiv  Detail & Related papers  (2024-09-17T06:18:47Z)
• EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation [54.32133648259802]
  We present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge. Building upon the video-language two-tower model and leveraging our meticulously organized egocentric video data, we introduce a novel foundation model called EgoVideo. This model is specifically designed to cater to the unique characteristics of egocentric videos and provides strong support for our competition submissions.
  arXiv  Detail & Related papers  (2024-06-26T05:01:37Z)
• Object Aware Egocentric Online Action Detection [23.504280692701272]
  We introduce an Object-Aware Module that integrates egocentric-specific priors into existing Online Action Detection frameworks. Our work can be seamlessly integrated into existing models with minimal overhead and bring consistent performance enhancements.
  arXiv  Detail & Related papers  (2024-06-03T07:58:40Z)
• Retrieval-Augmented Egocentric Video Captioning [53.2951243928289]
  EgoInstructor is a retrieval-augmented multimodal captioning model that automatically retrieves semantically relevant third-person instructional videos. We train the cross-view retrieval module with a novel EgoExoNCE loss that pulls egocentric and exocentric video features closer by aligning them to shared text features that describe similar actions.
  arXiv  Detail & Related papers  (2024-01-01T15:31:06Z)
• EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding [90.9111678470214]
  We propose EgoDistill, a distillation-based approach that learns to reconstruct heavy egocentric video clip features. Our method leads to significant improvements in efficiency, requiring 200x fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPICKitchens datasets, where our method outperforms state-of-the-art efficient video understanding methods.
  arXiv  Detail & Related papers  (2023-01-05T18:39:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.