Rethinking Position Embedding as a Context Controller for Multi-Reference and Multi-Shot Video Generation

• URL: http://arxiv.org/abs/2604.03738v1
• Date: Sat, 04 Apr 2026 13:50:38 GMT
• Title: Rethinking Position Embedding as a Context Controller for Multi-Reference and Multi-Shot Video Generation
• Abstract summary: We introduce PoCo (Position Embedding as a Context Controller), which incorporates position encoding as additional context control beyond semantic retrieval.<n>Building on PoCo, we develop a multi-reference and multi-shot video generation model capable of reliably controlling characters with extremely similar visual traits.
• Score: 6.953982005915382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent proprietary models such as Sora2 demonstrate promising progress in generating multi-shot videos conditioned on multiple reference characters. However, academic research on this problem remains limited. We study this task and identify a core challenge: when reference images exhibit highly similar appearances, the model often suffers from reference confusion, where semantically similar tokens degrade the model's ability to retrieve the correct context. To address this, we introduce PoCo (Position Embedding as a Context Controller), which incorporates position encoding as additional context control beyond semantic retrieval. By employing side information of tokens, PoCo enables precise token-level matching while preserving implicit semantic consistency modeling. Building on PoCo, we develop a multi-reference and multi-shot video generation model capable of reliably controlling characters with extremely similar visual traits. Extensive experiments demonstrate that PoCo improves cross-shot consistency and reference fidelity compared with various baselines.

Related papers

• ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships [53.24877477312131]
  ReBind is a systematic framework that introduces semantic instructions with embedded reference tokens.<n>We develop ReBind-Instruct, a specialized MLLM that learns to establish explicit bindings between visual attributes and their reference sources.<n>We also develop ReBind-Edit, which enables lightweight adaptation of text-to-video models to coordinate multiple references.
  arXiv  Detail & Related papers  (2026-07-16T07:43:27Z)
• Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment [20.811984469709508]
  Aura is a unified framework for high-fidelity and identity-consistent video generation.<n>We introduce AI director-level captions that provide dense and structured descriptions of video content.<n>We build a high-quality video-subject image dataset through a dedicated data construction pipeline.
  arXiv  Detail & Related papers  (2026-07-05T13:54:33Z)
• Self-Corrected Image Generation with Explainable Latent Rewards [55.29175717238288]
  We propose xLARD, a self-correcting framework that guides generation through Explainable LAtent RewarDs.<n>xLARD introduces a lightweight corrector that refines latent representations based on structured feedback from model-generated references.<n> Experiments show that xLARD improves semantic alignment and visual fidelity while maintaining generative priors.
  arXiv  Detail & Related papers  (2026-03-26T02:59:35Z)
• Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning [27.48318501076437]
  Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints.<n>We introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy.<n>We demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models.
  arXiv  Detail & Related papers  (2026-03-25T12:52:32Z)
• MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models [89.89575486159795]
  We introduce textbfMICON-Bench, a benchmark for multi-image context generation.<n>We propose an MLLM-driven Evaluation-by-Checkpoint framework for automatic verification of semantic and visual consistency.<n>We also present textbfDynamic Attention Rebalancing (DAR), a training-free, plug-and-play mechanism that dynamically adjusts attention during inference to enhance coherence and reduce hallucinations.
  arXiv  Detail & Related papers  (2026-02-23T04:32:52Z)
• AlcheMinT: Fine-grained Temporal Control for Multi-Reference Consistent Video Generation [58.844504598618094]
  We propose AlcheMinT, a unified framework that introduces explicit timestamps conditioning for subject-driven video generation.<n>Our approach introduces a novel positional encoding mechanism that unlocks the encoding of temporal intervals, associated in our case with subject identities.<n>We incorporate subject-descriptive text tokens to strengthen binding between visual identity and video captions, mitigating ambiguity during generation.
  arXiv  Detail & Related papers  (2025-12-11T18:59:34Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation [38.21587139161503]
  $textbfMultiBanana$ is designed to assess the edge of model capabilities by widely covering multi-reference-specific problems at scale.<n>Our analysis reveals their superior performances, typical failure modes, and areas for improvement.<n>MultiBanana will be released as an open benchmark to push the boundaries and establish a standardized basis for fair comparison in multi-reference image generation.
  arXiv  Detail & Related papers  (2025-11-28T08:49:55Z)
• Kaleido: Open-Sourced Multi-Subject Reference Video Generation Model [38.79676648965641]
  We present Kaleido, a subject-to-video(S2V) generation framework, which aims to synthesize subject-consistent videos conditioned on multiple reference images of target subjects.<n>Kaleido significantly outperforms previous methods in consistency, fidelity, and generalization, marking an advance in S2V generation.
  arXiv  Detail & Related papers  (2025-10-21T12:28:14Z)
• TokenAR: Multiple Subject Generation via Autoregressive Token-level enhancement [87.82338951215131]
  TokenAR is a simple but effective token-level enhancement mechanism to address reference identity confusion problem.<n>Instruct Token Injection plays as a role of extra visual feature container to inject detailed and complementary priors for reference tokens.<n>The identity-token disentanglement strategy (ITD) explicitly guides the token representations toward independently representing the features of each identity.
  arXiv  Detail & Related papers  (2025-10-18T03:36:26Z)
• Coreference as an indicator of context scope in multimodal narrative [3.9248546555042356]
  We show that large multimodal language models differ substantially from humans in the distribution of coreferential expressions.<n>We introduce a number of metrics to quantify the characteristics of coreferential patterns in both human- and machine-written texts.
  arXiv  Detail & Related papers  (2025-03-07T10:23:22Z)
• Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytelling [81.69474860607542]
  We present Openstory++, a large-scale dataset combining additional instance-level annotations with both images and text. We also present Cohere-Bench, a pioneering benchmark framework for evaluating the image generation tasks when long multimodal context is provided.
  arXiv  Detail & Related papers  (2024-08-07T11:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.