Locality-Aware Zero-Shot Human-Object Interaction Detection

• URL: http://arxiv.org/abs/2505.19503v1
• Date: Mon, 26 May 2025 04:31:34 GMT
• Title: Locality-Aware Zero-Shot Human-Object Interaction Detection
• Authors: Sanghyun Kim, Deunsol Jung, Minsu Cho,
• Abstract summary: We present LAIN, a novel zero-shot Human-Object Interaction (HOI) detection framework.<n>By infusing locality and interaction awareness into CLIP representation, LAIN captures detailed information about the human-object pairs.<n>Our experiments show that LAIN outperforms previous methods on various zero-shot settings.
• Score: 43.13461922628371
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent methods for zero-shot Human-Object Interaction (HOI) detection typically leverage the generalization ability of large Vision-Language Model (VLM), i.e., CLIP, on unseen categories, showing impressive results on various zero-shot settings. However, existing methods struggle to adapt CLIP representations for human-object pairs, as CLIP tends to overlook fine-grained information necessary for distinguishing interactions. To address this issue, we devise, LAIN, a novel zero-shot HOI detection framework enhancing the locality and interaction awareness of CLIP representations. The locality awareness, which involves capturing fine-grained details and the spatial structure of individual objects, is achieved by aggregating the information and spatial priors of adjacent neighborhood patches. The interaction awareness, which involves identifying whether and how a human is interacting with an object, is achieved by capturing the interaction pattern between the human and the object. By infusing locality and interaction awareness into CLIP representation, LAIN captures detailed information about the human-object pairs. Our extensive experiments on existing benchmarks show that LAIN outperforms previous methods on various zero-shot settings, demonstrating the importance of locality and interaction awareness for effective zero-shot HOI detection.

Related papers

• Funnel-HOI: Top-Down Perception for Zero-Shot HOI Detection [3.656114607436271]
  Human-object interaction detection (HOID) refers to localizing interactive human-object pairs in images and identifying the interactions.<n>We build a top-down framework named Funnel-HOI inspired by the human tendency to grasp well-defined concepts first and then associate them with abstract concepts during scene understanding.<n>A novel asymmetric co-attention mechanism mines these cues utilizing multimodal information (incorporating zero-shot capabilities) and yields stronger interaction representations at the encoder level.
  arXiv  Detail & Related papers  (2025-07-16T20:47:24Z)
• Visual-Geometric Collaborative Guidance for Affordance Learning [63.038406948791454]
  We propose a visual-geometric collaborative guided affordance learning network that incorporates visual and geometric cues. Our method outperforms the representative models regarding objective metrics and visual quality.
  arXiv  Detail & Related papers  (2024-10-15T07:35:51Z)
• Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection [70.96299509159981]
  Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction. Traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner.
  arXiv  Detail & Related papers  (2023-12-04T08:02:59Z)
• InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wild [40.489171608114574]
  Existing methods rely on frame-based detectors to locate interacting objects. We propose to leverage hand-object interaction to track interactive objects. Our proposed method outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-06T09:09:17Z)
• A Skeleton-aware Graph Convolutional Network for Human-Object Interaction Detection [14.900704382194013]
  We propose a skeleton-aware graph convolutional network for human-object interaction detection, named SGCN4HOI. Our network exploits the spatial connections between human keypoints and object keypoints to capture their fine-grained structural interactions via graph convolutions. It fuses such geometric features with visual features and spatial configuration features obtained from human-object pairs.
  arXiv  Detail & Related papers  (2022-07-11T15:20:18Z)
• Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition [111.87412719773889]
  We propose a joint learning framework for "interacted object localization" and "human action recognition" based on skeleton data. Our method achieves the best or competitive performance with the state-of-the-art methods for human action recognition.
  arXiv  Detail & Related papers  (2021-10-28T10:09:34Z)
• DRG: Dual Relation Graph for Human-Object Interaction Detection [65.50707710054141]
  We tackle the challenging problem of human-object interaction (HOI) detection. Existing methods either recognize the interaction of each human-object pair in isolation or perform joint inference based on complex appearance-based features. In this paper, we leverage an abstract spatial-semantic representation to describe each human-object pair and aggregate the contextual information of the scene via a dual relation graph.
  arXiv  Detail & Related papers  (2020-08-26T17:59:40Z)
• Learning Human-Object Interaction Detection using Interaction Points [140.0200950601552]
  We propose a novel fully-convolutional approach that directly detects the interactions between human-object pairs. Our network predicts interaction points, which directly localize and classify the inter-action. Experiments are performed on two popular benchmarks: V-COCO and HICO-DET.
  arXiv  Detail & Related papers  (2020-03-31T08:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.