LayerFlow: A Unified Model for Layer-aware Video Generation

• URL: http://arxiv.org/abs/2506.04228v1
• Date: Wed, 04 Jun 2025 17:59:58 GMT
• Title: LayerFlow: A Unified Model for Layer-aware Video Generation
• Abstract summary: Given per-layer prompts, LayerFlow generates videos for the transparent foreground, clean background, and blended scene.<n>It also supports versatile variants like decomposing a blended video or generating the background for the given foreground and vice versa.
• Score: 30.252873238218324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present LayerFlow, a unified solution for layer-aware video generation. Given per-layer prompts, LayerFlow generates videos for the transparent foreground, clean background, and blended scene. It also supports versatile variants like decomposing a blended video or generating the background for the given foreground and vice versa. Starting from a text-to-video diffusion transformer, we organize the videos for different layers as sub-clips, and leverage layer embeddings to distinguish each clip and the corresponding layer-wise prompts. In this way, we seamlessly support the aforementioned variants in one unified framework. For the lack of high-quality layer-wise training videos, we design a multi-stage training strategy to accommodate static images with high-quality layer annotations. Specifically, we first train the model with low-quality video data. Then, we tune a motion LoRA to make the model compatible with static frames. Afterward, we train the content LoRA on the mixture of image data with high-quality layered images along with copy-pasted video data. During inference, we remove the motion LoRA thus generating smooth videos with desired layers.

Related papers

• Explicit Layer Modeling for Video Object Insertion and Layer Decomposition [18.022394461124]
  We introduce TriLayer, where the foreground layers include both object appearance and associated visual effects.<n>This explicit supervision enables models to learn layered video representations directly rather than inferring them implicitly.<n>We propose DBL-Diffusion, a dual-branch diffusion framework that jointly models RGB composites and RGBA foreground layers.
  arXiv  Detail & Related papers  (2026-07-28T14:48:30Z)
• Vera: A Layered Diffusion Model for Content-Preserving Video Editing [31.27751797441043]
  We introduce Vera, a layered diffusion framework for content-preserving video editing.<n>Instead of regenerating the entire video, Vera generates an edit layer along with an alpha matte for compositing with the source video.<n>Vera outperforms leading open-source video editing models in content preservation while remaining competitive in edit quality, using 486K frames of layered training data.
  arXiv  Detail & Related papers  (2026-06-22T17:11:11Z)
• LaDe: Unified Multi-Layered Graphic Media Generation and Decomposition [4.794064830289192]
  We propose LaDe, a latent diffusion framework that generates a flexible number of semantically meaningful layers.<n>LaDe combines three components: an LLM-based prompt expander that transforms a short user intent into structured per-layer descriptions.<n>By conditioning on layer samples during training, our unified framework supports three tasks: text-to-image generation, text-to-layers media design generation, and media design decomposition.
  arXiv  Detail & Related papers  (2026-03-18T17:34:07Z)
• Qwen-Image-Layered: Towards Inherent Editability via Layer Decomposition [73.43121650616804]
  We propose textbfQwen-Image-Layered, an end-to-end diffusion model that decomposes a single RGB image into multiple semantically disentangled RGBA layers.<n>Our method significantly surpasses existing approaches in decomposition quality and establishes a new paradigm for consistent image editing.
  arXiv  Detail & Related papers  (2025-12-17T17:12:42Z)
• Layer-Aware Video Composition via Split-then-Merge [55.12521724893102]
  Split-then-Merge (StM) is a framework designed to enhance control in generative video composition.<n>StM splits a large corpus of unlabeled videos into dynamic foreground and background layers, then self-composes them to learn how dynamic subjects interact with diverse scenes.
  arXiv  Detail & Related papers  (2025-11-25T19:53:15Z)
• PSDiffusion: Harmonized Multi-Layer Image Generation via Layout and Appearance Alignment [24.964578950380947]
  PSDiffusion is a unified diffusion framework for simultaneous multi-layer text-to-image generation.<n>Our model can automatically generate multi-layer images with one RGB background and multiple RGBA foregrounds.<n>Our method introduces a global-layer interactive mechanism that generates layered-images concurrently and collaboratively.
  arXiv  Detail & Related papers  (2025-05-16T17:23:35Z)
• LayerAnimate: Layer-level Control for Animation [46.73168828036286]
  Existing anime generation video methods treat animation as a distinct data domain different from real-world videos, lacking fine-grained control at the layer level.<n>We introduce LayerAnimate, a novel video diffusion framework with layer-aware architecture that empowers the manipulation of layers through layer-level controls.<n>This framework opens up new possibilities for layer-level animation applications and creative flexibility.
  arXiv  Detail & Related papers  (2025-01-14T18:22:21Z)
• LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model [70.14953942532621]
  Layer-collaborative diffusion model, named LayerDiff, is designed for text-guided, multi-layered, composable image synthesis. Our model can generate high-quality multi-layered images with performance comparable to conventional whole-image generation methods. LayerDiff enables a broader range of controllable generative applications, including layer-specific image editing and style transfer.
  arXiv  Detail & Related papers  (2024-03-18T16:28:28Z)
• VideoStudio: Generating Consistent-Content and Multi-Scene Videos [88.88118783892779]
  VideoStudio is a framework for consistent-content and multi-scene video generation. VideoStudio leverages Large Language Models (LLM) to convert the input prompt into comprehensive multi-scene script. VideoStudio outperforms the SOTA video generation models in terms of visual quality, content consistency, and user preference.
  arXiv  Detail & Related papers  (2024-01-02T15:56:48Z)
• Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation [92.55296042611886]
  We propose a framework called "Reuse and Diffuse" dubbed $textitVidRD$ to produce more frames following the frames already generated by an LDM. We also propose a set of strategies for composing video-text data that involve diverse content from multiple existing datasets.
  arXiv  Detail & Related papers  (2023-09-07T08:12:58Z)
• Text2Layer: Layered Image Generation using Latent Diffusion Model [12.902259486204898]
  We propose to generate layered images from a layered image generation perspective. To achieve layered image generation, we train an autoencoder that is able to reconstruct layered images. Experimental results show that the proposed method is able to generate high-quality layered images.
  arXiv  Detail & Related papers  (2023-07-19T06:56:07Z)
• WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction [82.79642869586587]
  WALDO is a novel approach to the prediction of future video frames from past ones. Individual images are decomposed into multiple layers combining object masks and a small set of control points. The layer structure is shared across all frames in each video to build dense inter-frame connections.
  arXiv  Detail & Related papers  (2022-11-25T18:59:46Z)
• Unsupervised Video Interpolation by Learning Multilayered 2.5D Motion Fields [75.81417944207806]
  This paper presents a self-supervised approach to video frame learning that requires only a single video. We parameterize the video motions by solving an ordinary differentiable equation (ODE) defined on a time-varying motion field. This implicit neural representation learns the video as a space-time continuum, allowing frame-time continuum at any temporal resolution.
  arXiv  Detail & Related papers  (2022-04-21T06:17:05Z)
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis [73.82857768949651]
  We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled.
  arXiv  Detail & Related papers  (2021-04-30T15:38:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.