Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection

• URL: http://arxiv.org/abs/2312.01713v1
• Date: Mon, 4 Dec 2023 08:02:59 GMT
• Title: Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection
• Authors: Xubin Zhong, Changxing Ding, Yupeng Hu, Dacheng Tao
• Abstract summary: Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction. Traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner.
• Score: 70.96299509159981
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction; however, the interaction representations obtained using this method are entangled and lack interpretability. In contrast, traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner. In this paper, we improve the performance of one-stage methods by enabling them to extract disentangled interaction representations. First, we propose Shunted Cross-Attention (SCA) to extract human appearance, object appearance, and global context features using different cross-attention heads. This is achieved by imposing different masks on the cross-attention maps produced by the different heads. Second, we introduce the Interaction-aware Pose Estimation (IPE) task to learn interaction-relevant human pose features using a disentangled decoder. This is achieved with a novel attention module that accurately captures the human keypoints relevant to the current interaction category. Finally, our approach fuses the appearance feature and pose feature via element-wise addition to form the interaction representation. Experimental results show that our approach can be readily applied to existing one-stage HOI detectors. Moreover, we achieve state-of-the-art performance on two benchmarks: HICO-DET and V-COCO.

Related papers

• Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models [59.611697856666304]
  Human-object interaction (HOI) detection aims at detecting human-object pairs and predicting their interactions. We propose three prompts with VLM to generate human-centric visual cues within an image from multiple perspectives of humans. We develop a transformer-based multimodal fusion module with multitower architecture to integrate visual cue features into the instance and interaction decoders.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• HODN: Disentangling Human-Object Feature for HOI Detection [51.48164941412871]
  We propose a Human and Object Disentangling Network (HODN) to model the Human-Object Interaction (HOI) relationships explicitly. Considering that human features are more contributive to interaction, we propose a Human-Guide Linking method to make sure the interaction decoder focuses on the human-centric regions. Our proposed method achieves competitive performance on both the V-COCO and the HICO-Det Linking datasets.
  arXiv  Detail & Related papers  (2023-08-20T04:12:50Z)
• Knowledge Guided Bidirectional Attention Network for Human-Object Interaction Detection [3.0915392100355192]
  We argue that the independent use of the bottom-up parsing strategy in HOI is counter-intuitive and could lead to the diffusion of attention. We introduce a novel knowledge-guided top-down attention into HOI, and propose to model the relation parsing as a "look and search" process. We implement the process via unifying the bottom-up and top-down attention in a single encoder-decoder based model.
  arXiv  Detail & Related papers  (2022-07-16T16:42:49Z)
• Effective Actor-centric Human-object Interaction Detection [20.564689533862524]
  We propose a novel actor-centric framework to detect Human-Object Interaction in images. Our method achieves the state-of-the-art on the challenging V-COCO and HICO-DET benchmarks.
  arXiv  Detail & Related papers  (2022-02-24T10:24:44Z)
• HOTR: End-to-End Human-Object Interaction Detection with Transformers [26.664864824357164]
  We present a novel framework, referred to by HOTR, which directly predicts a set of human, object, interaction> triplets from an image. Our proposed algorithm achieves the state-of-the-art performance in two HOI detection benchmarks with an inference time under 1 ms after object detection.
  arXiv  Detail & Related papers  (2021-04-28T10:10:29Z)
• DRG: Dual Relation Graph for Human-Object Interaction Detection [65.50707710054141]
  We tackle the challenging problem of human-object interaction (HOI) detection. Existing methods either recognize the interaction of each human-object pair in isolation or perform joint inference based on complex appearance-based features. In this paper, we leverage an abstract spatial-semantic representation to describe each human-object pair and aggregate the contextual information of the scene via a dual relation graph.
  arXiv  Detail & Related papers  (2020-08-26T17:59:40Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)
• Learning Human-Object Interaction Detection using Interaction Points [140.0200950601552]
  We propose a novel fully-convolutional approach that directly detects the interactions between human-object pairs. Our network predicts interaction points, which directly localize and classify the inter-action. Experiments are performed on two popular benchmarks: V-COCO and HICO-DET.
  arXiv  Detail & Related papers  (2020-03-31T08:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.