MoCha:End-to-End Video Character Replacement without Structural Guidance

• URL: http://arxiv.org/abs/2601.08587v2
• Date: Wed, 14 Jan 2026 02:17:20 GMT
• Title: MoCha:End-to-End Video Character Replacement without Structural Guidance
• Authors: Zhengbo Xu, Jie Ma, Ziheng Wang, Zhan Peng, Jun Liang, Jing Li,
• Abstract summary: MoCha is a framework for replacing video characters with a user-provided identity.<n>We introduce a condition-aware RoPE and employ an RL-based post-training stage.<n>We design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs.
• Score: 14.573557179926079
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha

Related papers

• Fake-in-Facext: Towards Fine-Grained Explainable DeepFake Analysis [53.733003768566455]
  We propose the Fake-in-Facext (FiFa) framework, with contributions focusing on data annotation and model construction.<n>We first define a Facial Image Concept Tree (FICT) to divide facial images into fine-grained regional concepts.<n>Based on this dedicated data annotation, we introduce a novel Artifact-Grounding Explanation (AGE) task.<n>We propose a unified multi-task learning architecture, FiFa-MLLM, to simultaneously support abundant multimodal inputs and outputs for fine-grained Explainable DeepFake Analysis.
  arXiv  Detail & Related papers  (2025-10-23T13:16:12Z)
• From Large Angles to Consistent Faces: Identity-Preserving Video Generation via Mixture of Facial Experts [69.44297222099175]
  We introduce a Mixture of Facial Experts (MoFE) that captures distinct but mutually reinforcing aspects of facial attributes.<n>To mitigate dataset limitations, we have tailored a data processing pipeline centered on two key aspects: Face Constraints and Identity Consistency.<n>We have curated and refined a Large Face Angles (LFA) dataset from existing open-source human video datasets.
  arXiv  Detail & Related papers  (2025-08-13T04:10:16Z)
• Generative Video Matting [57.186684844156595]
  Video matting has traditionally been limited by the lack of high-quality ground-truth data.<n>Most existing video matting datasets provide only human-annotated imperfect alpha and foreground annotations.<n>We introduce a novel video matting approach that can effectively leverage the rich priors from pre-trained video diffusion models.
  arXiv  Detail & Related papers  (2025-08-11T12:18:55Z)
• OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions [77.04071342405055]
  We develop an Image-Video Transfer Mixed (IVTM) training with image editing data to enable instructive editing for the subject in the customized video.<n>We also propose a diffusion Transformer framework, OmniVCus, with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned Embedding (TAE)<n>Our method significantly surpasses state-of-the-art methods in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-06-29T18:43:00Z)
• Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-mask-based Embedding Router [72.29811385678168]
  We introduce Bind-Your-Avatar, an MM-DiT-based model specifically designed for multi-talking-character video generation in the same scene.<n>Specifically, we propose a novel framework incorporating a fine-grained Embedding Router that binds who' and speak what' together to address the audio-to-character correspondence control.
  arXiv  Detail & Related papers  (2025-06-24T17:50:16Z)
• MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement [47.064467920954776]
  We introduce MAGREF, a unified and effective framework for any-reference video generation.<n>Our approach incorporates masked guidance and a subject disentanglement mechanism.<n>Experiments on a comprehensive benchmark demonstrate that MAGREF consistently outperforms existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-05-29T17:58:15Z)
• Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer [25.39030226963548]
  We introduce the first application of a pretrained transformer-based video generative model for portrait animation.<n>Our method is validated through experiments on benchmark and newly proposed wild datasets.
  arXiv  Detail & Related papers  (2024-12-01T08:54:30Z)
• MegaScenes: Scene-Level View Synthesis at Scale [69.21293001231993]
  Scene-level novel view synthesis (NVS) is fundamental to many vision and graphics applications. We create a large-scale scene-level dataset from Internet photo collections, called MegaScenes, which contains over 100K structure from motion (SfM) reconstructions from around the world. We analyze failure cases of state-of-the-art NVS methods and significantly improve generation consistency.
  arXiv  Detail & Related papers  (2024-06-17T17:55:55Z)
• ReliTalk: Relightable Talking Portrait Generation from a Single Video [62.47116237654984]
  ReliTalk is a novel framework for relightable audio-driven talking portrait generation from monocular videos. Our key insight is to decompose the portrait's reflectance from implicitly learned audio-driven facial normals and images.
  arXiv  Detail & Related papers  (2023-09-05T17:59:42Z)
• Revealing Occlusions with 4D Neural Fields [19.71277637485384]
  For computer vision systems to operate in dynamic situations, they need to be able to represent and reason about object permanence. We introduce a framework for learning to estimate 4D visual representations from monocular-Dtemporal.
  arXiv  Detail & Related papers  (2022-04-22T20:14:42Z)
• Reference-Aided Part-Aligned Feature Disentangling for Video Person Re-Identification [18.13546384207381]
  We propose a textbfReference-textbfAided textbfPart-textbfAligned (textbfRAPA) framework to disentangle robust features of different parts. By using both modules, the informative parts of pedestrian in videos are well aligned and more discriminative feature representation is generated.
  arXiv  Detail & Related papers  (2021-03-21T06:53:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.