Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection

• URL: http://arxiv.org/abs/2604.02071v1
• Date: Thu, 02 Apr 2026 14:01:58 GMT
• Title: Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection
• Abstract summary: Human-Object Interaction (HOI) detection aims to localize human-object pairs and classify their interactions from a single image.<n>Recent approaches have leveraged Vision-Language Models (VLMs) to introduce semantic priors, significantly improving HOI detection performance.<n>We propose the Instance-centric Context Mining Network (InCoM-Net) to integrate rich semantic knowledge extracted from VLM with instance-specific features produced by an object detector.
• Score: 29.69451620133701
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Human-Object Interaction (HOI) detection aims to localize human-object pairs and classify their interactions from a single image, a task that demands strong visual understanding and nuanced contextual reasoning. Recent approaches have leveraged Vision-Language Models (VLMs) to introduce semantic priors, significantly improving HOI detection performance. However, existing methods often fail to fully capitalize on the diverse contextual cues distributed across the entire scene. To overcome these limitations, we propose the Instance-centric Context Mining Network (InCoM-Net)-a novel framework that effectively integrates rich semantic knowledge extracted from VLMs with instance-specific features produced by an object detector. This design enables deeper interaction reasoning by modeling relationships not only within each detected instance but also across instances and their surrounding scene context. InCoM-Net comprises two core components: Instancecentric Context Refinement (ICR), which separately extracts intra-instance, inter-instance, and global contextual cues from VLM-derived features, and Progressive Context Aggregation (ProCA), which iteratively fuses these multicontext features with instance-level detector features to support high-level HOI reasoning. Extensive experiments on the HICO-DET and V-COCO benchmarks show that InCoM-Net achieves state-of-the-art performance, surpassing previous HOI detection methods. Code is available at https://github.com/nowuss/InCoM-Net.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mamba [86.47790050206306]
  RefAVA++ comprises >2.9 million frames and >75.1k annotated persons in total.<n> RefAtomNet++ advances cross-modal token aggregation through a multi-hierarchical semantic-aligned cross-attention mechanism.<n>Experiments show that RefAtomNet++ establishes new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-10-18T10:41:19Z)
• Contextualized Representation Learning for Effective Human-Object Interaction Detection [17.242400169885453]
  Human-Object Interaction (HOI) detection aims to simultaneously localize human-object pairs and recognize their interactions.<n>We introduce a Contextualized Representation Learning that integrates both affordance-guided reasoning and contextual prompts.<n>Our proposed method demonstrates superior performance on both the HICO-Det and V-COCO datasets in most scenarios.
  arXiv  Detail & Related papers  (2025-09-16T08:03:16Z)
• ContextHOI: Spatial Context Learning for Human-Object Interaction Detection [24.381821663963898]
  spatial contexts are considered critical in Human-Object Interaction (HOI) recognition.<n>We present a dual-branch framework named ContextHOI, which efficiently captures both object detection features and spatial contexts.<n> ContextHOI achieves state-of-the-art performance on the HICO-DET and v-coco benchmarks.
  arXiv  Detail & Related papers  (2024-12-12T08:21:19Z)
• Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection [37.57355457749918]
  We introduce a novel framework for zero-shot HOI detection using Conditional Multi-Modal Prompts, namely CMMP. Unlike traditional prompt-learning methods, we propose learning decoupled vision and language prompts for interactiveness-aware visual feature extraction. Experiments demonstrate the efficacy of our detector with conditional multi-modal prompts, outperforming previous state-of-the-art on unseen classes of various zero-shot settings.
  arXiv  Detail & Related papers  (2024-08-05T14:05:25Z)
• Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model [3.3772986620114387]
  We introduce ISA-HOI, which extensively leverages knowledge from CLIP, aligning interactive semantics between visual and textual features. Our method achieves competitive results on the HICO-DET and V-COCO benchmarks with much fewer training epochs, and outperforms the state-of-the-art under zero-shot settings.
  arXiv  Detail & Related papers  (2024-04-19T07:24:32Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• Exploiting Scene Graphs for Human-Object Interaction Detection [81.49184987430333]
  Human-Object Interaction (HOI) detection is a fundamental visual task aiming at localizing and recognizing interactions between humans and objects. We propose a novel method to exploit this information, through the scene graph, for the Human-Object Interaction (SG2HOI) detection task. Our method, SG2HOI, incorporates the SG information in two ways: (1) we embed a scene graph into a global context clue, serving as the scene-specific environmental context; and (2) we build a relation-aware message-passing module to gather relationships from objects' neighborhood and transfer them into interactions.
  arXiv  Detail & Related papers  (2021-08-19T09:40:50Z)
• Exploiting Multi-Object Relationships for Detecting Adversarial Attacks in Complex Scenes [51.65308857232767]
  Vision systems that deploy Deep Neural Networks (DNNs) are known to be vulnerable to adversarial examples. Recent research has shown that checking the intrinsic consistencies in the input data is a promising way to detect adversarial attacks. We develop a novel approach to perform context consistency checks using language models.
  arXiv  Detail & Related papers  (2021-08-19T00:52:10Z)
• GID-Net: Detecting Human-Object Interaction with Global and Instance Dependency [67.95192190179975]
  We introduce a two-stage trainable reasoning mechanism, referred to as GID block. GID-Net is a human-object interaction detection framework consisting of a human branch, an object branch and an interaction branch. We have compared our proposed GID-Net with existing state-of-the-art methods on two public benchmarks, including V-COCO and HICO-DET.
  arXiv  Detail & Related papers  (2020-03-11T11:58:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.