Identity-Aware Human-Object Interaction Motion Captioning

• URL: http://arxiv.org/abs/2608.20690v1
• Date: Fri, 21 Aug 2026 02:53:26 GMT
• Title: Identity-Aware Human-Object Interaction Motion Captioning
• Abstract summary: We introduce Identity-Aware Human-Object Interaction Motion Captioning task.<n>This task requires each generated caption to specify both the subject identity and the corresponding HOI motion.<n>We propose ID-HOINet, which learns from multi-view videos while supporting single-view identity-aware HOI motion caption generation.
• Score: 14.724174102790839
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing human-object interaction (HOI) motion captioning methods typically describe what happens while referring to the subject using generic terms such as "a person" or "someone", without grounding the caption in subject identity. To address this limitation, we introduce Identity-Aware Human-Object Interaction Motion Captioning task. This task requires each generated caption to specify both the subject identity and the corresponding HOI motion. For example, the model generates "Sub_ID lifts the chair" rather than "A person lifts the chair". For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. We further propose ID-HOINet, which learns from multi-view videos while supporting single-view identity-aware HOI motion caption generation. ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR). MVIML learns from multi-view videos by modeling dependencies across temporal stages and camera viewpoints, capturing identity and interaction motion features. At inference, the TSCR first retrieves the subject identity and generates identity-agnostic HOI motion captions. TSCR then rewrites these captions with the predicted identity to produce the final identity-aware HOI motion captions. Experiments demonstrate that ID-HOINet achieves state-of-the-art performance. Code will be released upon acceptance.

Related papers

• GroupVideo: Multi-Identity Customized Text-to-Video Generation [53.89036635198158]
  GroupVideo is a novel framework that leverages multiple individual photographs to generate identitycustomized video.<n>GroupVideo incorporates multimodal identity alignment, while semantic alignment introduces a semantic perceiver to enhance the naturalness of motions.<n>In response to the shortage of multi-ID video datasets, we have curated a comprehensive high-quality dataset of 20,000 videos.
  arXiv  Detail & Related papers  (2026-07-23T08:09:04Z)
• Vera: Identity-Faithful Human Subject-to-Video Generation [66.45205582225942]
  We propose Vera, a unified human-centric S2V framework for single- and multi-person generation.<n>We first construct a million-pair identity-aligned human image-video dataset through person-level cross-clip retrieval.<n>We show that Vera improves human identity consistency, multi-person subject binding, and motion naturalness, while reducing identity confusion and excessive reference-image copying.
  arXiv  Detail & Related papers  (2026-07-22T15:08:30Z)
• Customizing Video Portraits via Identity-ActionDecoupling [32.57069491938664]
  Identity-Preserving Text-to-Video Generation (IPT2V) seeks to synthesize a temporally coherent video from a reference image and a textual description.<n>We introduce Identity-Action Decoupling framework as well as two loss function Identity Decoupling Loss and Text Alignment Loss to solve this problem.
  arXiv  Detail & Related papers  (2026-06-21T05:56:28Z)
• Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning [66.61493163603339]
  Existing sports video captioning methods often focus on the action yet overlook player identities, limiting their applicability.<n>This paper proposes a player-centric multimodal prompt generation network for identity-aware sports video captioning (LLM-IAVC)<n>We construct a new benchmark called NBA-Identity, a large identity-aware basketball video captioning dataset with 9,726 videos covering 9 major event types.
  arXiv  Detail & Related papers  (2025-07-27T07:30:56Z)
• ID-EA: Identity-driven Text Enhancement and Adaptation with Textual Inversion for Personalized Text-to-Image Generation [33.84646269805187]
  ID-EA is a novel framework that guides text embeddings to align with visual identity embeddings.<n> ID-EA substantially outperforms state-of-the-art methods in identity preservation metrics.<n>It generates personalized portraits 15 times faster than existing approaches.
  arXiv  Detail & Related papers  (2025-07-16T07:42:02Z)
• Proteus-ID: ID-Consistent and Motion-Coherent Video Customization [17.792780924370103]
  Video identity customization seeks to synthesize realistic, temporally coherent videos of a specific subject, given a single reference image and a text prompt.<n>This task presents two core challenges: maintaining identity consistency while aligning with the described appearance and actions, and generating natural, fluid motion without unrealistic stiffness.<n>We introduce Proteus-ID, a novel diffusion-based framework for identity-consistent and motion-coherent video customization.
  arXiv  Detail & Related papers  (2025-06-30T11:05:32Z)
• MICap: A Unified Model for Identity-aware Movie Descriptions [16.287294191608893]
  We present a new single stage approach that can seamlessly switch between id-aware caption generation or FITB when given a caption with blanks. Our model, Movie-Identity Captioner (MICap), uses a shared auto-regressive decoder that benefits from training with FITB and full-caption generation objectives.
  arXiv  Detail & Related papers  (2024-05-19T08:54:12Z)
• Magic-Me: Identity-Specific Video Customized Diffusion [72.05925155000165]
  We propose a controllable subject identity controllable video generation framework, termed Video Custom Diffusion (VCD) With a specified identity defined by a few images, VCD reinforces the identity characteristics and injects frame-wise correlation for stable video outputs. We conducted extensive experiments to verify that VCD is able to generate stable videos with better ID over the baselines.
  arXiv  Detail & Related papers  (2024-02-14T18:13:51Z)
• SOVC: Subject-Oriented Video Captioning [59.04029220586337]
  We propose a new video captioning task, Subject-Oriented Video Captioning (SOVC), which aims to allow users to specify the describing target via a bounding box. To support this task, we construct two subject-oriented video captioning datasets based on two widely used video captioning datasets.
  arXiv  Detail & Related papers  (2023-12-20T17:44:32Z)
• Identity-Aware Multi-Sentence Video Description [105.13845996039277]
  We introduce an auxiliary task of Fill-in the Identity, that aims to predict persons' IDs consistently within a set of clips. One of the key components is a gender-aware textual representation as well an additional gender prediction objective in the main model. Experiments show that our proposed Fill-in the Identity model is superior to several baselines and recent works.
  arXiv  Detail & Related papers  (2020-08-22T09:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.