Short-term Object Interaction Anticipation with Disentangled Object Detection @ Ego4D Short Term Object Interaction Anticipation Challenge

• URL: http://arxiv.org/abs/2407.05713v1
• Date: Mon, 8 Jul 2024 08:13:16 GMT
• Title: Short-term Object Interaction Anticipation with Disentangled Object Detection @ Ego4D Short Term Object Interaction Anticipation Challenge
• Authors: Hyunjin Cho, Dong Un Kang, Se Young Chun,
• Abstract summary: Short-term object interaction anticipation is an important task in egocentric video analysis. Our proposed method, SOIA-DOD, effectively decomposes it into 1) detecting active object and 2) classifying interaction and predicting their timing. Our method first detects all potential active objects in the last frame of egocentric video by fine-tuning a pre-trained YOLOv9.
• Score: 11.429137967096935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Short-term object interaction anticipation is an important task in egocentric video analysis, including precise predictions of future interactions and their timings as well as the categories and positions of the involved active objects. To alleviate the complexity of this task, our proposed method, SOIA-DOD, effectively decompose it into 1) detecting active object and 2) classifying interaction and predicting their timing. Our method first detects all potential active objects in the last frame of egocentric video by fine-tuning a pre-trained YOLOv9. Then, we combine these potential active objects as query with transformer encoder, thereby identifying the most promising next active object and predicting its future interaction and time-to-contact. Experimental results demonstrate that our method outperforms state-of-the-art models on the challenge test set, achieving the best performance in predicting next active objects and their interactions. Finally, our proposed ranked the third overall top-5 mAP when including time-to-contact predictions. The source code is available at https://github.com/KeenyJin/SOIA-DOD.

Related papers

• Object-centric Video Representation for Long-term Action Anticipation [33.115854386196126]
  Key motivation is that objects provide important cues to recognize and predict human-object interactions. We propose to build object-centric video representations by leveraging visual-language pretrained models. To recognize and predict human-object interactions, we use a Transformer-based neural architecture.
  arXiv  Detail & Related papers  (2023-10-31T22:54:31Z)
• HODN: Disentangling Human-Object Feature for HOI Detection [51.48164941412871]
  We propose a Human and Object Disentangling Network (HODN) to model the Human-Object Interaction (HOI) relationships explicitly. Considering that human features are more contributive to interaction, we propose a Human-Guide Linking method to make sure the interaction decoder focuses on the human-centric regions. Our proposed method achieves competitive performance on both the V-COCO and the HICO-Det Linking datasets.
  arXiv  Detail & Related papers  (2023-08-20T04:12:50Z)
• Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videos [31.620555223890626]
  We study the problem of Short-Term Object interaction anticipation (STA) We propose NAOGAT, a multi-modal end-to-end transformer network, to guide the model to predict context-aware future actions. Our model outperforms existing methods on two separate datasets.
  arXiv  Detail & Related papers  (2023-08-16T12:07:02Z)
• Enhancing Next Active Object-based Egocentric Action Anticipation with Guided Attention [45.60789439017625]
  Short-term action anticipation (STA) in first-person videos is a challenging task. We propose a novel approach that applies a guided attention mechanism between objects. Our method, GANO, is a multi-modal, end-to-end, single transformer-based network.
  arXiv  Detail & Related papers  (2023-05-22T11:56:10Z)
• StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation [14.188006024550257]
  We study the short-term object interaction anticipation problem from the egocentric point of view. Our approach simultaneously processes a still image and a video detecting and localizing next-active objects. Our method is ranked first in the public leaderboard of the EGO4D short term object interaction anticipation challenge 2022.
  arXiv  Detail & Related papers  (2023-04-08T09:01:37Z)
• Anticipating Next Active Objects for Egocentric Videos [29.473527958651317]
  This paper addresses the problem of anticipating the next-active-object location in the future, for a given egocentric video clip. We propose a transformer-based self-attention framework to identify and locate the next-active-object in an egocentric clip.
  arXiv  Detail & Related papers  (2023-02-13T13:44:52Z)
• H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions [62.510951695174604]
  "Hypothesize, Simulate, Act, Update, and Repeat" (H-SAUR) is a probabilistic generative framework that generates hypotheses about how objects articulate given input observations. We show that the proposed model significantly outperforms the current state-of-the-art articulated object manipulation framework. We further improve the test-time efficiency of H-SAUR by integrating a learned prior from learning-based vision models.
  arXiv  Detail & Related papers  (2022-10-22T18:39:33Z)
• Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions [82.90906153293585]
  We propose a graph neural network, HO-GCN, to fuse motion data and dynamic descriptors for the prediction task. We show the proposed network that consumes dynamic descriptors can achieve state-of-the-art prediction results and help the network better generalize to unseen objects.
  arXiv  Detail & Related papers  (2022-06-25T09:55:39Z)
• Suspected Object Matters: Rethinking Model's Prediction for One-stage Visual Grounding [93.82542533426766]
  We propose a Suspected Object Transformation mechanism (SOT) to encourage the target object selection among the suspected ones. SOT can be seamlessly integrated into existing CNN and Transformer-based one-stage visual grounders. Extensive experiments demonstrate the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2022-03-10T06:41:07Z)
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter [56.00554240240515]
  INVIGORATE is a robot system that interacts with human through natural language and grasps a specified object in clutter. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. We build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules.
  arXiv  Detail & Related papers  (2021-08-25T07:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.