Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning

• URL: http://arxiv.org/abs/2507.20163v1
• Date: Sun, 27 Jul 2025 07:30:56 GMT
• Title: Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning
• Authors: Zeyu Xi, Haoying Sun, Yaofei Wu, Junchi Yan, Haoran Zhang, Lifang Wu, Liang Wang, Changwen Chen,
• Abstract summary: Existing sports video captioning methods often focus on the action yet overlook player identities, limiting their applicability.<n>This paper proposes a player-centric multimodal prompt generation network for identity-aware sports video captioning (LLM-IAVC)<n>We construct a new benchmark called NBA-Identity, a large identity-aware basketball video captioning dataset with 9,726 videos covering 9 major event types.
• Score: 66.61493163603339
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing sports video captioning methods often focus on the action yet overlook player identities, limiting their applicability. Although some methods integrate extra information to generate identity-aware descriptions, the player identities are sometimes incorrect because the extra information is independent of the video content. This paper proposes a player-centric multimodal prompt generation network for identity-aware sports video captioning (LLM-IAVC), which focuses on recognizing player identities from a visual perspective. Specifically, an identity-related information extraction module (IRIEM) is designed to extract player-related multimodal embeddings. IRIEM includes a player identification network (PIN) for extracting visual features and player names, and a bidirectional semantic interaction module (BSIM) to link player features with video content for mutual enhancement. Additionally, a visual context learning module (VCLM) is designed to capture the key video context information. Finally, by integrating the outputs of the above modules as the multimodal prompt for the large language model (LLM), it facilitates the generation of descriptions with player identities. To support this work, we construct a new benchmark called NBA-Identity, a large identity-aware basketball video captioning dataset with 9,726 videos covering 9 major event types. The experimental results on NBA-Identity and VC-NBA-2022 demonstrate that our proposed model achieves advanced performance. Code and dataset are publicly available at https://github.com/Zeyu1226-mt/LLM-IAVC.

Related papers

• Proteus-ID: ID-Consistent and Motion-Coherent Video Customization [17.792780924370103]
  Video identity customization seeks to synthesize realistic, temporally coherent videos of a specific subject, given a single reference image and a text prompt.<n>This task presents two core challenges: maintaining identity consistency while aligning with the described appearance and actions, and generating natural, fluid motion without unrealistic stiffness.<n>We introduce Proteus-ID, a novel diffusion-based framework for identity-consistent and motion-coherent video customization.
  arXiv  Detail & Related papers  (2025-06-30T11:05:32Z)
• IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes [20.662082715151886]
  We introduce a new dataset termed MultiClip-Bench, featuring dense descriptions and instruction-based question-answering pairs tailored for multi-shot scenarios.<n>We then contribute a new model IPFormer-VideoLLM, which injection of instance-level features as instance prompts through an efficient attention-based connector.
  arXiv  Detail & Related papers  (2025-06-26T09:30:57Z)
• IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model [52.697180472760635]
  This paper explores the potential of character identities memory and recognition across multiple visual scenarios. We propose visual instruction tuning with ID reference and develop an ID-Aware Large Vision-Language Model, IDA-VLM. Our research introduces a novel benchmark MM-ID, to examine LVLMs on instance IDs memory and recognition across four dimensions.
  arXiv  Detail & Related papers  (2024-07-10T12:11:59Z)
• Domain-Guided Masked Autoencoders for Unique Player Identification [62.87054782745536]
  Masked autoencoders (MAEs) have emerged as a superior alternative to conventional feature extractors. Motivated by human vision, we devise a novel domain-guided masking policy for MAEs termed d-MAE. We conduct experiments on three large-scale sports datasets.
  arXiv  Detail & Related papers  (2024-03-17T20:14:57Z)
• Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark [49.54265459763042]
  We construct a multimodal basketball game knowledge graph (KG_NBA_2022) to provide additional knowledge beyond videos. Then, a dataset that contains 9 types of fine-grained shooting events and 286 players' knowledge is constructed based on KG_NBA_2022. We develop a knowledge guided entity-aware video captioning network (KEANet) based on a candidate player list in encoder-decoder form for basketball live text broadcast.
  arXiv  Detail & Related papers  (2024-01-25T02:08:37Z)
• Multi-Stage Spatio-Temporal Aggregation Transformer for Video Person Re-identification [78.08536797239893]
  We propose a novel Multi-Stage Spatial-Temporal Aggregation Transformer (MSTAT) with two novel designed proxy embedding modules. MSTAT consists of three stages to encode the attribute-associated, the identity-associated, and the attribute-identity-associated information from the video clips. We show that MSTAT can achieve state-of-the-art accuracies on various standard benchmarks.
  arXiv  Detail & Related papers  (2023-01-02T05:17:31Z)
• Identity-Aware Multi-Sentence Video Description [105.13845996039277]
  We introduce an auxiliary task of Fill-in the Identity, that aims to predict persons' IDs consistently within a set of clips. One of the key components is a gender-aware textual representation as well an additional gender prediction objective in the main model. Experiments show that our proposed Fill-in the Identity model is superior to several baselines and recent works.
  arXiv  Detail & Related papers  (2020-08-22T09:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.