Vorch-IR: Long-Form Unified Multimodal Identity Replacement Video Generation

• URL: http://arxiv.org/abs/2608.05648v1
• Date: Thu, 06 Aug 2026 06:46:35 GMT
• Title: Vorch-IR: Long-Form Unified Multimodal Identity Replacement Video Generation
• Abstract summary: Vorch-IR is a unified framework that supports single- and dual-person identity replacement, with optional background replacement, in a single model.<n>Built on LTX2, Vorch-IR jointly conditions on a driving video, indexed reference images, and a textual editing instruction.<n>Experiments using automatic metrics and pairwise human evaluation demonstrate strong identity preservation, motion fidelity, and temporal coherence across diverse scenarios.
• Score: 28.493796603648175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video identity replacement seeks to transfer the identities of one or more subjects while preserving the motion, expressions, and temporal structure of a driving video. Existing methods largely target single-person settings and often require task-specific structural controls, such as masks or pose representations, limiting their flexibility in general multimodal editing systems. Progress on multi-person replacement is further constrained by the scarcity of paired training data. We present Vorch-IR, a unified framework that supports single- and dual-person identity replacement, with optional background replacement, in a single model. Built on LTX2, Vorch-IR jointly conditions on a driving video, indexed reference images, and a textual editing instruction. The reference images need not match the pose, layout, or spatial configuration of the driving video: their roles as subject or background references are specified through the instruction. Dense visual conditions are fused through self-attention, while a vision-language context establishes semantic correspondence through cross-attention. We further develop an automatic data construction pipeline that synthesizes paired supervision for all four editing settings. Experiments using automatic metrics and pairwise human evaluation demonstrate strong identity preservation, motion fidelity, and temporal coherence across diverse scenarios. A temporal overlapping inference strategy additionally extends the short-clip model to minute-long generation without autoregressive continuation.

Related papers

• Vorch-Omni: Multi-Task Orchestration of Sight and Sound [51.39284238118125]
  Vorch- Omni is a unified framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-language formulation.<n>It flexibly treats video and audio signals as either conditioning inputs or generation targets.<n>It supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video transformation, and audio-visual editing.
  arXiv  Detail & Related papers  (2026-08-06T09:38:30Z)
• GroupVideo: Multi-Identity Customized Text-to-Video Generation [53.89036635198158]
  GroupVideo is a novel framework that leverages multiple individual photographs to generate identitycustomized video.<n>GroupVideo incorporates multimodal identity alignment, while semantic alignment introduces a semantic perceiver to enhance the naturalness of motions.<n>In response to the shortage of multi-ID video datasets, we have curated a comprehensive high-quality dataset of 20,000 videos.
  arXiv  Detail & Related papers  (2026-07-23T08:09:04Z)
• Keyframe-Anchored Identity Preservation for Sequential-Action Video Generation [10.89407870318018]
  The IPVG26 challenge extends this framework from a single holistic prompt to a temporally structured specification.<n>This temporal structure presents a challenge not encountered in previous identity-preserving generation tasks.<n>We address this challenge with a training-free, three-stage pipeline framework.
  arXiv  Detail & Related papers  (2026-07-20T14:16:49Z)
• Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment [20.811984469709508]
  Aura is a unified framework for high-fidelity and identity-consistent video generation.<n>We introduce AI director-level captions that provide dense and structured descriptions of video content.<n>We build a high-quality video-subject image dataset through a dedicated data construction pipeline.
  arXiv  Detail & Related papers  (2026-07-05T13:54:33Z)
• AlcheMinT: Fine-grained Temporal Control for Multi-Reference Consistent Video Generation [58.844504598618094]
  We propose AlcheMinT, a unified framework that introduces explicit timestamps conditioning for subject-driven video generation.<n>Our approach introduces a novel positional encoding mechanism that unlocks the encoding of temporal intervals, associated in our case with subject identities.<n>We incorporate subject-descriptive text tokens to strengthen binding between visual identity and video captions, mitigating ambiguity during generation.
  arXiv  Detail & Related papers  (2025-12-11T18:59:34Z)
• MultiShotMaster: A Controllable Multi-Shot Video Generation Framework [67.38203939500157]
  Current generation techniques excel at single-shot clips but struggle to produce narrative multi-shot videos.<n>We propose MultiShotMaster, a framework for highly controllable multi-shot video generation.
  arXiv  Detail & Related papers  (2025-12-02T18:59:48Z)
• CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion [62.04833878126661]
  We tackle the dual challenges of video understanding and controllable video generation within a unified diffusion framework.<n>We propose CtrlVDiff, a unified diffusion model trained with a Hybrid Modality Control Strategy (HMCS) that routes and fuses features from depth, normals, segmentation, edges, and graphics-based intrinsics (albedo, roughness, metallic)<n>Across understanding and generation benchmarks, CtrlVDiff delivers superior controllability and fidelity, enabling layer-wise edits (relighting, material adjustment, object insertion) and surpassing state-of-the-art baselines while remaining robust when some modalities are unavailable.
  arXiv  Detail & Related papers  (2025-11-26T07:27:11Z)
• ID-Composer: Multi-Subject Video Synthesis with Hierarchical Identity Preservation [48.59900036213667]
  Video generative models pretrained on large-scale datasets can produce high-quality videos, but are often conditioned on text or a single image.<n>We introduce ID-Composer, a novel framework that tackles multi-subject video generation from a text prompt and reference images.
  arXiv  Detail & Related papers  (2025-11-01T11:29:14Z)
• Proteus-ID: ID-Consistent and Motion-Coherent Video Customization [17.792780924370103]
  Video identity customization seeks to synthesize realistic, temporally coherent videos of a specific subject, given a single reference image and a text prompt.<n>This task presents two core challenges: maintaining identity consistency while aligning with the described appearance and actions, and generating natural, fluid motion without unrealistic stiffness.<n>We introduce Proteus-ID, a novel diffusion-based framework for identity-consistent and motion-coherent video customization.
  arXiv  Detail & Related papers  (2025-06-30T11:05:32Z)
• Identity-Aware Multi-Sentence Video Description [105.13845996039277]
  We introduce an auxiliary task of Fill-in the Identity, that aims to predict persons' IDs consistently within a set of clips. One of the key components is a gender-aware textual representation as well an additional gender prediction objective in the main model. Experiments show that our proposed Fill-in the Identity model is superior to several baselines and recent works.
  arXiv  Detail & Related papers  (2020-08-22T09:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.