Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation

• URL: http://arxiv.org/abs/2507.05963v2
• Date: Wed, 09 Jul 2025 07:01:34 GMT
• Title: Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation
• Authors: Zhenghao Zhang, Junchao Liao, Xiangyu Meng, Long Qin, Weizhi Wang,
• Abstract summary: Tora is a diffusion transformer model for motion-guided video generation.<n>Tora2 introduces several design improvements to expand its capabilities in both appearance and motion customization.<n>Tora2 is the first method to achieve simultaneous multi-entity customization of appearance and motion for video generation.
• Score: 8.108805590363392
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in diffusion transformer models for motion-guided video generation, such as Tora, have shown significant progress. In this paper, we present Tora2, an enhanced version of Tora, which introduces several design improvements to expand its capabilities in both appearance and motion customization. Specifically, we introduce a decoupled personalization extractor that generates comprehensive personalization embeddings for multiple open-set entities, better preserving fine-grained visual details compared to previous methods. Building on this, we design a gated self-attention mechanism to integrate trajectory, textual description, and visual information for each entity. This innovation significantly reduces misalignment in multimodal conditioning during training. Moreover, we introduce a contrastive loss that jointly optimizes trajectory dynamics and entity consistency through explicit mapping between motion and personalization embeddings. Tora2 is, to our best knowledge, the first method to achieve simultaneous multi-entity customization of appearance and motion for video generation. Experimental results demonstrate that Tora2 achieves competitive performance with state-of-the-art customization methods while providing advanced motion control capabilities, which marks a critical advancement in multi-condition video generation. Project page: https://ali-videoai.github.io/Tora2_page/.

Related papers

• Versatile Transition Generation with Image-to-Video Diffusion [89.67070538399457]
  We present a Versatile Transition video Generation framework that can generate smooth, high-fidelity, and semantically coherent video transitions.<n>We show that VTG achieves superior transition performance consistently across all four tasks.
  arXiv  Detail & Related papers  (2025-08-03T10:03:56Z)
• SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation [56.90807453045657]
  SynMotion is a motion-customized video generation model that jointly leverages semantic guidance and visual adaptation.<n>At the semantic level, we introduce the dual-em semantic comprehension mechanism which disentangles subject and motion representations.<n>At the visual level, we integrate efficient motion adapters into a pre-trained video generation model to enhance motion fidelity and temporal coherence.
  arXiv  Detail & Related papers  (2025-06-30T10:09:32Z)
• ATI: Any Trajectory Instruction for Controllable Video Generation [25.249489701215467]
  We propose a unified framework for motion control in video generation that seamlessly integrates camera movement, object-level translation, and fine-grained local motion.<n>Our approach offers a cohesive solution by projecting user-defined trajectories into the latent space of pre-trained image-to-video generation models.
  arXiv  Detail & Related papers  (2025-05-28T23:49:18Z)
• VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models [71.9811050853964]
  VideoJAM is a novel framework that instills an effective motion prior to video generators.<n>VideoJAM achieves state-of-the-art performance in motion coherence.<n>These findings emphasize that appearance and motion can be complementary and, when effectively integrated, enhance both the visual quality and the coherence of video generation.
  arXiv  Detail & Related papers  (2025-02-04T17:07:10Z)
• Prototypical Transformer as Unified Motion Learners [38.31482767855841]
  Prototypical Transformer (ProtoFormer) is a framework that approaches various motion tasks from a prototype perspective. ProtoFormer seamlessly integrates prototype learning with Transformer by thoughtfully considering motion dynamics.
  arXiv  Detail & Related papers  (2024-06-03T17:41:28Z)
• MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion [94.66090422753126]
  MotionFollower is a lightweight score-guided diffusion model for video motion editing. It delivers superior motion editing performance and exclusively supports large camera movements and actions. Compared with MotionEditor, the most advanced motion editing model, MotionFollower achieves an approximately 80% reduction in GPU memory.
  arXiv  Detail & Related papers  (2024-05-30T17:57:30Z)
• Animate Your Motion: Turning Still Images into Dynamic Videos [58.63109848837741]
  We introduce Scene and Motion Conditional Diffusion (SMCD), a novel methodology for managing multimodal inputs. SMCD incorporates a recognized motion conditioning module and investigates various approaches to integrate scene conditions. Our design significantly enhances video quality, motion precision, and semantic coherence.
  arXiv  Detail & Related papers  (2024-03-15T10:36:24Z)
• Compositional Transformers for Scene Generation [13.633811200719627]
  We introduce the GANformer2 model, an iterative object-oriented transformer, explored for the task of generative modeling. We show it achieves state-of-the-art performance in terms of visual quality, diversity and consistency. Further experiments demonstrate the model's disentanglement and provide a deeper insight into its generative process.
  arXiv  Detail & Related papers  (2021-11-17T08:11:42Z)
• Motion-Attentive Transition for Zero-Shot Video Object Segmentation [99.44383412488703]
  We present a Motion-Attentive Transition Network (MATNet) for zero-shot object segmentation. An asymmetric attention block, called Motion-Attentive Transition (MAT), is designed within a two-stream encoder. In this way, the encoder becomes deeply internative, allowing for closely hierarchical interactions between object motion and appearance.
  arXiv  Detail & Related papers  (2020-03-09T16:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.