FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

• URL: http://arxiv.org/abs/2609.05959v1
• Date: Sat, 05 Sep 2026 07:53:11 GMT
• Title: FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection
• Abstract summary: FineHOI is a zero-shot HOI framework that explicitly models interactions from dense patch-level features.<n>Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships.<n>Experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods.
• Score: 26.006274699996027
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Human-Object Interaction (HOI) detection aims to localize humans and objects in images and classify their interactions. Zero-shot HOI focuses on recognizing interactions that are not observed during training, requiring models to generalize beyond seen verb-object compositions. Recent approaches leverage Vision-Language Models (VLMs), benefiting from rich semantic representations. However, they often rely on global or detector-centric features that compress interaction cues and hinder fine-grained spatial reasoning. To overcome this limitation, we propose FineHOI, a zero-shot HOI framework that explicitly models interactions from dense patch-level features. Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships, which are not preserved by global and detector-centric representations. To this end, we introduce an Adaptive Part-Level Attention module that decomposes humans and objects into semantically coherent parts via unsupervised clustering, and re-weights them based on their interaction relevance. These representations are then integrated through a Region-Aware Interaction Transformer that integrates part-aware and global features and produces the final HOI embedding. Extensive experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods, achieving particularly strong gains on unseen interactions. Code is available at https://github.com/francescotonini/fine-hoi.

Related papers

• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• Learning Human-Object Interaction as Groups [52.28258599873394]
  GroupHOI is a framework that propagates contextual information in terms of geometric proximity and semantic similarity.<n>It exhibits leading performance on the more challenging Nonverbal Interaction Detection task.
  arXiv  Detail & Related papers  (2025-10-21T07:25:10Z)
• Contextualized Representation Learning for Effective Human-Object Interaction Detection [17.242400169885453]
  Human-Object Interaction (HOI) detection aims to simultaneously localize human-object pairs and recognize their interactions.<n>We introduce a Contextualized Representation Learning that integrates both affordance-guided reasoning and contextual prompts.<n>Our proposed method demonstrates superior performance on both the HICO-Det and V-COCO datasets in most scenarios.
  arXiv  Detail & Related papers  (2025-09-16T08:03:16Z)
• Funnel-HOI: Top-Down Perception for Zero-Shot HOI Detection [3.656114607436271]
  Human-object interaction detection (HOID) refers to localizing interactive human-object pairs in images and identifying the interactions.<n>We build a top-down framework named Funnel-HOI inspired by the human tendency to grasp well-defined concepts first and then associate them with abstract concepts during scene understanding.<n>A novel asymmetric co-attention mechanism mines these cues utilizing multimodal information (incorporating zero-shot capabilities) and yields stronger interaction representations at the encoder level.
  arXiv  Detail & Related papers  (2025-07-16T20:47:24Z)
• Locality-Aware Zero-Shot Human-Object Interaction Detection [43.13461922628371]
  We present LAIN, a novel zero-shot Human-Object Interaction (HOI) detection framework.<n>By infusing locality and interaction awareness into CLIP representation, LAIN captures detailed information about the human-object pairs.<n>Our experiments show that LAIN outperforms previous methods on various zero-shot settings.
  arXiv  Detail & Related papers  (2025-05-26T04:31:34Z)
• Visual-Geometric Collaborative Guidance for Affordance Learning [63.038406948791454]
  We propose a visual-geometric collaborative guided affordance learning network that incorporates visual and geometric cues. Our method outperforms the representative models regarding objective metrics and visual quality.
  arXiv  Detail & Related papers  (2024-10-15T07:35:51Z)
• RR-Net: Injecting Interactive Semantics in Human-Object Interaction Detection [40.65483058890176]
  Latest end-to-end HOI detectors are short of relation reasoning, which leads to inability to learn HOI-specific interactive semantics for predictions. We first present a progressive Relation-aware Frame, which brings a new structure and parameter sharing pattern for interaction inference. Based on modules above, we construct an end-to-end trainable framework named Relation Reasoning Network (abbr. RR-Net)
  arXiv  Detail & Related papers  (2021-04-30T14:03:10Z)
• ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection [101.56529337489417]
  We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of human, action, object> in images. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities.
  arXiv  Detail & Related papers  (2020-08-14T09:11:18Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)
• Learning Human-Object Interaction Detection using Interaction Points [140.0200950601552]
  We propose a novel fully-convolutional approach that directly detects the interactions between human-object pairs. Our network predicts interaction points, which directly localize and classify the inter-action. Experiments are performed on two popular benchmarks: V-COCO and HICO-DET.
  arXiv  Detail & Related papers  (2020-03-31T08:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.