QueryCraft: Transformer-Guided Query Initialization for Enhanced Human-Object Interaction Detection

• URL: http://arxiv.org/abs/2508.08590v1
• Date: Tue, 12 Aug 2025 03:11:16 GMT
• Title: QueryCraft: Transformer-Guided Query Initialization for Enhanced Human-Object Interaction Detection
• Abstract summary: We propose a novel plug-and-play HOI detection framework that incorporates semantic priors and guided feature learning.<n>Our method achieves state-of-the-art performance and strong generalizations on HICO-Det and V-COCO benchmarks.
• Score: 7.030364980618468
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions in images. Although DETR-based methods have recently emerged as the mainstream framework for HOI detection, they still suffer from a key limitation: Randomly initialized queries lack explicit semantics, leading to suboptimal detection performance. To address this challenge, we propose QueryCraft, a novel plug-and-play HOI detection framework that incorporates semantic priors and guided feature learning through transformer-based query initialization. Central to our approach is \textbf{ACTOR} (\textbf{A}ction-aware \textbf{C}ross-modal \textbf{T}ransf\textbf{OR}mer), a cross-modal Transformer encoder that jointly attends to visual regions and textual prompts to extract action-relevant features. Rather than merely aligning modalities, ACTOR leverages language-guided attention to infer interaction semantics and produce semantically meaningful query representations. To further enhance object-level query quality, we introduce a \textbf{P}erceptual \textbf{D}istilled \textbf{Q}uery \textbf{D}ecoder (\textbf{PDQD}), which distills object category awareness from a pre-trained detector to serve as object query initiation. This dual-branch query initialization enables the model to generate more interpretable and effective queries for HOI detection. Extensive experiments on HICO-Det and V-COCO benchmarks demonstrate that our method achieves state-of-the-art performance and strong generalization. Code will be released upon publication.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction [127.64173950476702]
  We study direct corpus interaction (DCI), where an agent searches the raw corpus directly with general-purpose terminal tools.<n>This approach requires no offline indexing and adapts naturally to evolving local corpora.<n>Across IR benchmarks and end-to-end agentic search tasks, this simple setup substantially outperforms strong sparse, dense, and reranking baselines.
  arXiv  Detail & Related papers  (2026-05-03T19:13:11Z)
• WeDetect: Fast Open-Vocabulary Object Detection as Retrieval [74.39703419628829]
  Open-vocabulary object detection aims to detect arbitrary classes via text prompts.<n> Methods without cross-modal fusion layers (non-fusion) offer faster inference by treating recognition as a retrieval problem.<n>We develop a model family named WeDetect to achieve state-of-the-art performance across 15 benchmarks with high inference efficiency.
  arXiv  Detail & Related papers  (2025-12-13T12:40:28Z)
• CIR-CoT: Towards Interpretable Composed Image Retrieval via End-to-End Chain-of-Thought Reasoning [93.05917922306196]
  Composed Image Retrieval (CIR) aims to find a target image from a reference image and a modification text.<n>CIR-CoT is the first end-to-end retrieval-oriented MLLM designed to integrate explicit Chain-of-Thought (CoT) reasoning.
  arXiv  Detail & Related papers  (2025-10-09T09:41:45Z)
• DQEN: Dual Query Enhancement Network for DETR-based HOI Detection [19.742290580421322]
  Human-Object Interaction (HOI) detection focuses on localizing human-object pairs and recognizing their interactions.<n>In DETR-based HOI models, queries with clear meaning are crucial for accurately detecting HOIs.<n>We propose a Dual Query Enhancement Network (DQEN) to enhance object and interaction queries.
  arXiv  Detail & Related papers  (2025-08-26T10:11:08Z)
• DS-Det: Single-Query Paradigm and Attention Disentangled Learning for Flexible Object Detection [39.56089737473775]
  We propose DS-Det, a more efficient transformer detector capable of detecting a flexible number of objects in images.<n>Specifically, we reformulate and introduce a new unified Single-Query paradigm for decoder modeling.<n>We also propose a simplified decoder framework through attention disentangled learning.
  arXiv  Detail & Related papers  (2025-07-26T05:40:04Z)
• METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection [25.542175004831844]
  Open-vocabulary video visual relationship detection aims to detect objects and their relationships in videos without being restricted by predefined object or relationship categories.<n>Existing methods leverage the rich semantic knowledge of pre-trained vision-language models such as CLIP to identify novel categories.<n>We propose Mutual EnhancemenT of Objects and Relationships (METOR) to jointly model and mutually enhance object detection and relationship classification in open-vocabulary scenarios.
  arXiv  Detail & Related papers  (2025-05-10T14:45:43Z)
• Knowledge Graph Completion with Relation-Aware Anchor Enhancement [50.50944396454757]
  We propose a relation-aware anchor enhanced knowledge graph completion method (RAA-KGC)<n>We first generate anchor entities within the relation-aware neighborhood of the head entity.<n>Then, by pulling the query embedding towards the neighborhoods of the anchors, it is tuned to be more discriminative for target entity matching.
  arXiv  Detail & Related papers  (2025-04-08T15:22:08Z)
• Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection [11.497620257835964]
  We propose CCKT-Det trained without any extra supervision.<n>The proposed framework constructs a cyclic and dynamic knowledge transfer from language queries and visual region features extracted from vision-language models (VLMs)<n> CCKT-Det can consistently improve performance as the scale of VLMs increases, all while requiring the detector at a moderate level of overhead.
  arXiv  Detail & Related papers  (2025-03-14T02:04:28Z)
• End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting [68.37943632270505]
  Open-vocabulary video visual relationship detection aims to expand video visual relationship detection beyond categories.<n>Existing methods usually use trajectory detectors trained on closed datasets to detect object trajectories.<n>We propose an open-vocabulary relationship that leverages the rich semantic knowledge of CLIP to discover novel relationships.
  arXiv  Detail & Related papers  (2024-09-19T06:25:01Z)
• Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection [37.57355457749918]
  We introduce a novel framework for zero-shot HOI detection using Conditional Multi-Modal Prompts, namely CMMP. Unlike traditional prompt-learning methods, we propose learning decoupled vision and language prompts for interactiveness-aware visual feature extraction. Experiments demonstrate the efficacy of our detector with conditional multi-modal prompts, outperforming previous state-of-the-art on unseen classes of various zero-shot settings.
  arXiv  Detail & Related papers  (2024-08-05T14:05:25Z)
• ESTextSpotter: Towards Better Scene Text Spotting with Explicit Synergy in Transformer [88.61312640540902]
  We introduce Explicit Synergy-based Text Spotting Transformer framework (ESTextSpotter) Our model achieves explicit synergy by modeling discriminative and interactive features for text detection and recognition within a single decoder. Experimental results demonstrate that our model significantly outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-20T03:22:23Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection [54.041049052843604]
  We present STEMD, a novel end-to-end framework that enhances the DETR-like paradigm for multi-frame 3D object detection. First, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network. Finally, it poses a challenge for the network to distinguish between the positive query and other highly similar queries that are not the best match.
  arXiv  Detail & Related papers  (2023-07-01T13:53:14Z)
• Category-Aware Transformer Network for Better Human-Object Interaction Detection [20.857034771924997]
  We study the issue of promoting transformer-based HOI detectors by initializing the Object Query with category-aware semantic information. Specifically, the Object Query would be represented via category priors represented by an external object detection model to yield better performance. A HOI detection model equipped with our idea outperforms the baseline by a large margin to achieve a new state-of-the-art result.
  arXiv  Detail & Related papers  (2022-04-11T07:21:24Z)
• Open-Vocabulary DETR with Conditional Matching [86.1530128487077]
  OV-DETR is an open-vocabulary detector based on DETR. It can detect any object given its class name or an exemplar image. It achieves non-trivial improvements over current state of the arts.
  arXiv  Detail & Related papers  (2022-03-22T16:54:52Z)
• Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics [6.678312249123534]
  We aim to boost end-to-end models with object-guided statistical priors. We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-guided hierarchy. The above modules combined composes Object-guided Cross-modal Network (OCN)
  arXiv  Detail & Related papers  (2022-02-01T07:39:04Z)
• End-to-End Object Detection with Transformers [88.06357745922716]
  We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss.
  arXiv  Detail & Related papers  (2020-05-26T17:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.