TransAnimate: Taming Layer Diffusion to Generate RGBA Video

• URL: http://arxiv.org/abs/2503.17934v1
• Date: Sun, 23 Mar 2025 04:27:46 GMT
• Title: TransAnimate: Taming Layer Diffusion to Generate RGBA Video
• Authors: Xuewei Chen, Zhimin Chen, Yiren Song,
• Abstract summary: TransAnimate is an innovative framework that integrates RGBA image generation techniques with video generation modules.<n>We introduce an interactive motion-guided control mechanism, where directional arrows define movement and colors adjust scaling.<n>We have developed a pipeline for creating an RGBA video dataset, incorporating high-quality game effect videos, extracted foreground objects, and synthetic transparent videos.
• Score: 3.7031943280491997
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-video generative models have made remarkable advancements in recent years. However, generating RGBA videos with alpha channels for transparency and visual effects remains a significant challenge due to the scarcity of suitable datasets and the complexity of adapting existing models for this purpose. To address these limitations, we present TransAnimate, an innovative framework that integrates RGBA image generation techniques with video generation modules, enabling the creation of dynamic and transparent videos. TransAnimate efficiently leverages pre-trained text-to-transparent image model weights and combines them with temporal models and controllability plugins trained on RGB videos, adapting them for controllable RGBA video generation tasks. Additionally, we introduce an interactive motion-guided control mechanism, where directional arrows define movement and colors adjust scaling, offering precise and intuitive control for designing game effects. To further alleviate data scarcity, we have developed a pipeline for creating an RGBA video dataset, incorporating high-quality game effect videos, extracted foreground objects, and synthetic transparent videos. Comprehensive experiments demonstrate that TransAnimate generates high-quality RGBA videos, establishing it as a practical and effective tool for applications in gaming and visual effects.

Related papers

• Versatile Transition Generation with Image-to-Video Diffusion [89.67070538399457]
  We present a Versatile Transition video Generation framework that can generate smooth, high-fidelity, and semantically coherent video transitions.<n>We show that VTG achieves superior transition performance consistently across all four tasks.
  arXiv  Detail & Related papers  (2025-08-03T10:03:56Z)
• RoboTransfer: Geometry-Consistent Video Diffusion for Robotic Visual Policy Transfer [33.178540405656676]
  RoboTransfer is a diffusion-based video generation framework for robotic data synthesis.<n>It integrates multi-view geometry with explicit control over scene components, such as background and object attributes.<n>RoboTransfer is capable of generating multi-view videos with enhanced geometric consistency and visual fidelity.
  arXiv  Detail & Related papers  (2025-05-29T07:10:03Z)
• VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation [62.64811405314847]
  We introduce VidCRAFT3, a novel framework for precise image-to-video generation.<n>It enables control over camera motion, object motion, and lighting direction simultaneously.<n>Experiments on benchmark datasets demonstrate the efficacy of VidCRAFT3 in producing high-quality video content.
  arXiv  Detail & Related papers  (2025-02-11T13:11:59Z)
• BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations [82.94002870060045]
  Existing video generation models struggle to follow complex text prompts and synthesize multiple objects.<n>We develop a blob-grounded video diffusion model named BlobGEN-Vid that allows users to control object motions and fine-grained object appearance.<n>We show that our framework is model-agnostic and build BlobGEN-Vid based on both U-Net and DiT-based video diffusion models.
  arXiv  Detail & Related papers  (2025-01-13T19:17:06Z)
• TransPixeler: Advancing Text-to-Video Generation with Transparency [43.6546902960154]
  We introduce TransPixeler, a method to extend pretrained video models for RGBA generation while retaining the original RGB capabilities.<n>Our approach effectively generates diverse and consistent RGBA videos, advancing the possibilities for VFX and interactive content creation.
  arXiv  Detail & Related papers  (2025-01-06T13:32:16Z)
• T-SVG: Text-Driven Stereoscopic Video Generation [87.62286959918566]
  This paper introduces the Text-driven Stereoscopic Video Generation (T-SVG) system.<n>It streamlines video generation by using text prompts to create reference videos.<n>These videos are transformed into 3D point cloud sequences, which are rendered from two perspectives with subtle parallax differences.
  arXiv  Detail & Related papers  (2024-12-12T14:48:46Z)
• Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer [25.39030226963548]
  We introduce the first application of a pretrained transformer-based video generative model for portrait animation.<n>Our method is validated through experiments on benchmark and newly proposed wild datasets.
  arXiv  Detail & Related papers  (2024-12-01T08:54:30Z)
• TVG: A Training-free Transition Video Generation Method with Diffusion Models [12.037716102326993]
  Transition videos play a crucial role in media production, enhancing the flow and coherence of visual narratives. Recent advances in diffusion model-based video generation offer new possibilities for creating transitions but face challenges such as poor inter-frame relationship modeling and abrupt content changes. We propose a novel training-free Transition Video Generation (TVG) approach using video-level diffusion models that addresses these limitations without additional training.
  arXiv  Detail & Related papers  (2024-08-24T00:33:14Z)
• Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion [8.068194154084967]
  This paper tackles a challenge of how to exert precise control over object motion for realistic video synthesis.<n>To accomplish this, we control object movements using bounding boxes and extend this control to the renderings of 2D or 3D boxes in pixel space.<n>Our method, Ctrl-V, leverages modified and fine-tuned Stable Video Diffusion (SVD) models to solve both trajectory and video generation.
  arXiv  Detail & Related papers  (2024-06-09T03:44:35Z)
• Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators [70.17041424896507]
  Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets. We propose a new task of zero-shot text-to-video generation using existing text-to-image synthesis methods. Our method performs comparably or sometimes better than recent approaches, despite not being trained on additional video data.
  arXiv  Detail & Related papers  (2023-03-23T17:01:59Z)
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis [73.82857768949651]
  We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled.
  arXiv  Detail & Related papers  (2021-04-30T15:38:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.