InstanceV: Instance-Level Video Generation

• URL: http://arxiv.org/abs/2511.23146v1
• Date: Fri, 28 Nov 2025 12:52:37 GMT
• Title: InstanceV: Instance-Level Video Generation
• Abstract summary: We propose InstanceV, a video generation framework that enables i) instance-level control and ii) global semantic consistency.<n>We introduce the Shared Timestep-Adaptive Prompt Enhancement module, which connects local instances with global semantics.<n>Finally, we propose a new benchmark, named InstanceBench, which combines general video quality metrics with instance-aware metrics.
• Score: 97.01147000740417
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in text-to-video diffusion models have enabled the generation of high-quality videos conditioned on textual descriptions. However, most existing text-to-video models rely solely on textual conditions, lacking general fine-grained controllability over video generation. To address this challenge, we propose InstanceV, a video generation framework that enables i) instance-level control and ii) global semantic consistency. Specifically, with the aid of proposed Instance-aware Masked Cross-Attention mechanism, InstanceV maximizes the utilization of additional instance-level grounding information to generate correctly attributed instances at designated spatial locations. To improve overall consistency, We introduce the Shared Timestep-Adaptive Prompt Enhancement module, which connects local instances with global semantics in a parameter-efficient manner. Furthermore, we incorporate Spatially-Aware Unconditional Guidance during both training and inference to alleviate the disappearance of small instances. Finally, we propose a new benchmark, named InstanceBench, which combines general video quality metrics with instance-aware metrics for more comprehensive evaluation on instance-level video generation. Extensive experiments demonstrate that InstanceV not only achieves remarkable instance-level controllability in video generation, but also outperforms existing state-of-the-art models in both general quality and instance-aware metrics across qualitative and quantitative evaluations.

Related papers

• InstanceRSR: Real-World Super-Resolution via Instance-Aware Representation Alignment [6.761263337778605]
  Super-resolution methods based on generative priors have achieved remarkable progress in producing high-quality and globally consistent reconstructions.<n>However, they often struggle to recover fine-grained details of diverse object instances in complex real-world scenes.<n>This limitation primarily arises because commonly adopted denoising losses (e.g., MSE) inherently favor global consistency while neglecting instance-level perception and restoration.
  arXiv  Detail & Related papers  (2026-03-25T12:27:58Z)
• BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration [56.98981194478512]
  We propose a unified framework that handles a broad range of subject-to-video scenarios.<n>We introduce an MLLM-DiT framework in which a pretrained multimodal large language model performs deep cross-modal reasoning to ground entities.<n>Experiments on the OpenS2V benchmark demonstrate that our method achieves superior performance across subject consistency, naturalness, and text relevance in generated videos.
  arXiv  Detail & Related papers  (2025-10-01T02:41:11Z)
• Improving Generalized Visual Grounding with Instance-aware Joint Learning [45.53531162436934]
  Generalized visual grounding tasks are designed to accommodate multi-target and non-target scenarios.<n>We propose InstanceVG, a framework equipped with instance-aware capabilities to tackle both GREC and GRES.<n>To instantiate the framework, we assign each instance query a prior reference point, which also serves as an additional basis for target matching.
  arXiv  Detail & Related papers  (2025-09-17T07:00:51Z)
• IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes [20.662082715151886]
  We introduce a new dataset termed MultiClip-Bench, featuring dense descriptions and instruction-based question-answering pairs tailored for multi-shot scenarios.<n>We then contribute a new model IPFormer-VideoLLM, which injection of instance-level features as instance prompts through an efficient attention-based connector.
  arXiv  Detail & Related papers  (2025-06-26T09:30:57Z)
• InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption [40.07877801495546]
  Training relies on video-caption paired data, which plays a crucial role in enhancing generation performance.<n>Current video captions often suffer from insufficient details, hallucinations and imprecise motion depiction.<n>We propose a novel instance-aware structured caption framework, termed InstanceCap, to achieve instance-level and fine-grained video caption.
  arXiv  Detail & Related papers  (2024-12-12T13:48:40Z)
• CAVIS: Context-Aware Video Instance Segmentation [12.71520768233772]
  We introduce the Context-Aware Video Instance (CAVIS), a novel framework designed to enhance instance association.<n>To efficiently extract and leverage this information, we propose the Context-Aware Instance Tracker (CAIT)<n>We design the Prototypical Cross-frame Contrastive (PCC) loss, which ensures consistency in object-level features across frames.
  arXiv  Detail & Related papers  (2024-07-03T11:11:16Z)
• TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation [97.96178992465511]
  We argue that generated videos should incorporate the emergence of new concepts and their relation transitions like in real-world videos as time progresses. To assess the Temporal Compositionality of video generation models, we propose TC-Bench, a benchmark of meticulously crafted text prompts, corresponding ground truth videos, and robust evaluation metrics.
  arXiv  Detail & Related papers  (2024-06-12T21:41:32Z)
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models [70.19437817951673]
  We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics.
  arXiv  Detail & Related papers  (2023-10-17T17:50:46Z)
• Less than Few: Self-Shot Video Instance Segmentation [50.637278655763616]
  We propose to automatically learn to find appropriate support videos given a query. We tackle, for the first time, video instance segmentation in a self-shot (and few-shot) setting. We provide strong baseline performances that utilize a novel transformer-based model.
  arXiv  Detail & Related papers  (2022-04-19T13:14:43Z)
• Reliable Shot Identification for Complex Event Detection via Visual-Semantic Embedding [72.9370352430965]
  We propose a visual-semantic guided loss method for event detection in videos. Motivated by curriculum learning, we introduce a negative elastic regularization term to start training the classifier with instances of high reliability. An alternative optimization algorithm is developed to solve the proposed challenging non-net regularization problem.
  arXiv  Detail & Related papers  (2021-10-12T11:46:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.